Lionbridge Machine Translation Tracker

We’ve enhanced the Lionbridge Machine Translation (MT) Quality Tracker report, given the prevalence and promise of generative AI (GenAI) / Large Language Models (LLMs). From here on in, the report will include GPT-4 translation results in addition to GPT-3.5 and Davinci results and, of course, Neural MT (NMT) engine performance.

What are some of our latest findings? Some noteworthy GPT-4 peculiarities.

We faced several issues associated with GPT-4, including slow performance, its inability to provide translations for various reasons, and inconsistent behavior, such as missing translations in some runs but not in others.

Finding #1 — GPT-4’s failure to translate some text.

GPT-4 failed to translate a particular sentence in our MT test set.

After some research, we determined that a term with a sexual connotation in particular contexts caused the issue. To be clear, the sentence in our test set was entirely standard and acceptable. Nonetheless, the term triggered GPT-4’s sexual content filter anyway, and the AI subsequently censored the translation of that sentence and outputted nothing. We were surprised by this result for two reasons:

• The typical use of that term in isolation had no issues.
• The context of that particular sentence had no problematic interpretation.

This observation led us to conclude that perhaps a part of the GPT-4 filtering mechanism was based on a simple forbidden word list that also includes ambiguous terms. This approach is problematic as it is prone to overfire and provoke false positives, which is a serious issue for professional translation.

Because earlier Machine Translation technologies, such as Neural MT engines, do not have this type of content filtering issue, we can conclude it is a limitation of LLM technology.

The limitation has implications for real-world scenarios. For instance, imagine you need to translate medical content associated with gynecology or sexual education. You may be surprised that the LLM will not translate some of your text.

Interestingly, this issue happened to us only when translating that sentence into a particular language, Chinese, but not when translating it into other languages. This result indicates that the filter was on the GPT-4 output. The solution is to turn off the content filters for translation tasks.

Finding #2 — GPT-4’s output variability.

We found LLM Machine Translation output highly variable after five weeks of tracking, particularly with GPT-4.

While we expected this outcome for generative AI, the variability was more significant than anticipated — even when we used Temperature and Top Probability (Top_p) parameter settings to reduce creativity and make the output more deterministic. The translation output was different in every single GPT run we conducted, even when we ran translations one right after the other.

Both translations may be acceptable even though they differ. Nonetheless, this is another aspect to control and another difference from the previous Neural MT paradigm.

We are starting to intuit that this potential change of paradigm — from NMT to LLM MT — may not only be a technological change but also require us to have a change in mindset: We may need to be prepared to live with less deterministic outputs, even when using the very same input and the very same parameters, and expect to see more variability than what we are used to with current automation.

While we may have to live with more uncertainty to some extent, it may be possible to use some mechanisms and best practices to make that variability somewhat controllable.

Finally, as you review our chart, note that the Edit Distance decrease for GPT-4 does not indicate decreasing quality. It is merely a reflection of the variability of GPT outputs. Next month, we may see the line going up. Watch this space for developments and more insight.