“How to you guarantee ChatGPT responses are accurate?” To understand, we decided to ask ChatGPT a question.A Forbes story warned that OpenAI’s amazing ChatGPT can create convincing human-like lies! This begs the question,
We then asked Richard Davies, our AI engineer at Guildhawk to verify them.
ChatGPT had already told us on New Years’ Eve what could go wrong with AI translations generated by ChatGPT. The chatbot listed four big risks that could result if you publish those translations without verifying them.
Richard pointed-out there was another big risk, ‘breach of copyright’. We now wanted to learn more about how you best verify results from ChatGPT.
Get more insights on ChatGPT security best practices: A comprehensive guide.
Here’s the question we asked ChatGPT
‘List the top ways to guarantee that translations created by ChatGPT are accurate, unbiased and not offensive.’
ChatGPT created this list of the top 10 ways to guarantee accuracy of GPT translation results:
- Use multiple language models and compare their output to ensure accuracy and reduce bias.
- Use human editors to proofread and validate the translations.
- Use a diverse set of training data to reduce bias in the model’s output.
- Use pre-processing methods to remove any offensive or biased language from the training data.
- Continuously monitor and evaluate the model’s output for accuracy and bias.
- Use external datasets to evaluate the model’s performance on specific language pairs.
- Use Sentiment analysis to check the overall tone of the translations
- Implement a feedback mechanism for users to report any issues with the translations.
- Using Additional NLP tools like Named Entity Recognition (NER) for checking translations consistency and named entities correctness.
- Regularly update the model with new data to improve its performance and reduce bias
Richard reviews the ChatGPT answer?
The answer generated by ChatGPT seems to be evidence based but is it completely accurate?
Richard reviewed the answers and gave his observations.
- Multiple languages: ChatGPT or GPT-3 doesn’t have multiple language models available. From reading the responses, ChatGPT seems to have misinterpreted the question. The response is more akin to asking how to increase accuracy and reduce bias in Machine Translation in general.
- Human editors: Yes, a reasonable response.
- Diverse data: You don’t get to decide the training data used to train ChatGPT or GPT-3, it’s a pre-trained model. In this case using appropriate and concise prompts would be more applicable answer.
- Pre-processing: This would be improved by reinforcing a rule earlier in the prompt that ensures no offensive words appear in the translation.
- Continuous monitoring: Yes, this seems reasonable.
- External datasets: You could use open-source external datasets to evaluate translation results of GPT before going ahead with using it in a business setting, so this is a good point.
- Sentiment analysis: This also makes sense, but it would be better to use a Machine Learning model trained specifically for sentiment analysis. A zero-shot model like GPT3 is a generalist model, a specialist model will outperform it.
- Feedback: Yes, this should be implemented in any product making use of large language models.
- Additional NLP: Not completely clear how Name Entity Recognition would check the consistency of translation or named entities correctness. Name entity only enables you to understand where a name entity (name, company name, date, etc) is within natural language. Thus, it is questionable how understanding where a name entity is helps with this.
- Update with new data: This would make sense, but remember GPT-3 is a pre-trained model. GPT users don’t have access to regularly update the model. It’s an in-house OpenAI product.
Based on Richard’s review of the ChatGPT answer, our conclusion is the chatbot response is pretty accurate. The result was generated quickly.
The 10 points are human-like and easy to understand, even for a non-AI trained person.
The OpenAI website login makes it perfectly clear that ChatGPT has limitations. And that results should not be used for things like technical translations or obtaining medical advice.
Recommendations for using GPT-3 for Business Translations
Richard’s personal recommendation is that GPT-3 alone should not be used for business translations due to the evidence provided. On the other hand, it may be appropriate to use for personal translations where accuracy isn’t extremely important.
That’s because relying on GPT-3 in business critical situations can cause real-world repercussions for businesses, especially in the legal and medical sector. GPT is a revolutionary tool and evolving fast, so watch this space.
Recommendations for using ChatGPT with multilingual Digital Human avatars
It is not recommended to use ChatGPT to translate scripts used by Digital Humans in training videos. This is due to the risk that the translation generated could be wrong.
New AI-powered Machine Translation tools like Guildhawk Aided can be used because this guarantees the best results. High quality is maintained since algorithms are trained on Guildhawk’s vast private data lake.
The Guildhawk dataset is completely accurate, unbiased since it was created over several years by expert linguists. It is also updated every day.
OpenAI faces surge of copyright lawsuits
The CBC, Toronto Star and Globe and Mail are seeking billions in damages. The lawsuit, filed in Ontario, is asking for punitive damages, a share of OpenAI’s profits and a ban on using their content in the future.
News Media Canada president Paul Deegan says OpenAI is “free-riding” on publishers’ investment in journalism.
They are asking for up to C$20,000 per article, which could be billions.
The company is accused of scraping and using proprietary content without permission to train its GPT models. This is part of a bigger trend of Canadian media vs American tech companies, including Meta.
US outlets like The New York Times have filed similar suits. OpenAI is worth $150bn and has licensing deals with some media companies but didn’t comment on the case.
- OpenAI's AI models rely on publicly available data, raising questions about the line between fair use and legal or ethical violations. Creators argue that their work is being exploited to develop billion-dollar technology without compensation.