Lionbridge Machine Translation Tracker

Introducing our assessment tool to help you choose the best machine translation engine for your needs

Introduction

Machine translation (MT) has been around for many years, but in recent years has been evolving at an exponential rate. As companies produce ever-growing amounts of content in different languages, they have come to see machine translation as a promise of extended reach in an increasingly globalized world. 

Companies seeking to deploy machine translation have four basic strategies to explore: 

Public MT

Use of services like Google or Bing Translate. Those services are readily available to everyone at no cost however they do not have more advanced functionalities like terminology customization and data might be re-used in other services. 

On-Premise MT

This approach assumes company will deploy machine translation server in their IT environments. This is a very secure, but comes at a significant cost, is complex to deploy and manage and requires ongoing maintenance.

Cloud MT

Works similar like the Public MT and is hosted in the cloud but creates a company dedicated instance. Any data shared with the service is tightly secured and not shared with 3rd parties. It provides additional capabilities around terminology customization and other benefits but can result in vendor lock in and less then optimal quality of machine translation across multiple language pairs. 

Best of Breed MT

Is a single platform that allows to manage multiple machine translation engines, providing single layer of terminology customization, easy to manage interface and ability to choose best engines for different language pairs and types of content. 

No matter which strategy they choose - picking the right engine may be hard without proper data and experience. Lionbridge is an expert in machine translation with more than two decades of experience and has gathered a large volume of linguistic and quality data about machine translation technology that can help make the right choice. This webpage will provide basic information about performance of machine translation engines for the most common language pairs to help companies make best choices for their content. 

Want to learn more about different types of machine translation technologies? Check out our blog Machine Translation in Translation.

Lionbridge Machine Translation Tracker Methodology

Lionbridge uses inverse edit distance as a scoring method. Edit distance is a measure of discrepancy among translations that uses the number of characters (for Asian languages) or words (Western language) that need to be changed by a human post editor as a measure of quality. The higher the metric the better the quality.

Out of the four machine translation engines we assessed, Google and Bing NMT demonstrate the best performance across different language pairs and for general content. However, in certain language combinations, more specialized engines perform best: DeepL has the strongest performance in German and Amazon best translates Chinese.

Evaluating Overall MT Performance

Chart 1: Comparison of average quality of top Machine Translation engines for German, Russian, Spanish and Chinese. 

Each machine translation engine performs differently depending on number of factors, like the quality of the source content, the source language and target languages, subject matter and if the machine translation engine was trained. In overall quality comparison, as of January 2021, Google NMT is still leading in long-term performance for all languages. Bing and Amazon are the MT engines that have experienced highest fluctuations in their quality scores over last 12 months, with significant spikes in quality.

Per Language Pair Quality

Chart 2: Comparison of average quality of top Machine Translation engines for specific language pairs.

Some machine translation engines excel at certain language pairs. Looking at long term language quality performance data, the best MT engine for German was DeepL, while Amazon has distanced all the competition for Chinese. Google has been in the lead for most of the other top languages, closely followed by Bing. 

Per Domain Performance

Chart 3: Comparison of average quality of top Machine Translation engines for specific domain.

Our research has shown that for Automotive and Machinery domain, untrained DeepL MT engine was consistently best performing engine for top evaluated languages. Amazon appeared to be best choice for the e-commerce and retail sectors. Neural machine translation engines can also be trained in language for a certain industry that would further improve quality. 

For more insights and future trends about machine translation, read our Future of Language Tech blog post – Future of Machine Translation.

Disclaimer

  1. Machine Translation engines in this report are assessed monthly by Lionbridge.
  2. The data provided is for illustration purposes and each case should be treated and assessed individually.
  3. This report is generated based on source data preselected by Lionbridge Machine Translation teams. The same source data is submitted to every machine translation engine and language pair each time, making comparisons between translation engines possible.
  4. No customer data has been used in the generation of the report.

Contact Us