The Trials and Tribulations of Predicting Post-Editing Productivity

The Trials and Tribulations of Predicting Machine Translation Post- Editing Productivity Lena Marg Welocalize, Inc. Frederick, MD, United States E-mail: lena.marg@welocalize.com Abstract While an increasing number of (automatic) metrics is available to assess the linguistic quality of machine translations, their interpretation remains cryptic to many users, specifically in the translation community. They are clearly useful for indicating certain overarching trends, but say little about actual improvements for translation buyers or post-editors. However, these metrics are commonly referenced when discussing pricing and models, both with translation buyers and service providers. With the aim of focusing on automatic metrics that are easier to understand for non-research users, we identified Edit Distance (or Post-Edit Distance) as a good fit. While Edit Distance as such does not express cognitive effort or time spent editing machine translation suggestions, we found that it correlates strongly with the productivity tests we performed, for various language pairs and domains. This paper aims to analyse Edit Distance and productivity data on a segment level based on data gathered over some years. Drawing from these findings, we want to then explore how Edit Distance could help in predicting productivity on new content. Some further analysis is proposed, with findings to be presented at the conference. Keywords: Machine Translation, Metrics, Quality Evaluation, Edit Distance, Post-Edit Productivity, Predicting Productivity 1. Introduction There is an increasing number of metrics available to assess the linguistic quality produced by machine translation (MT) systems. Using several of them in parallel helps to validate the individual scores and thus increase confidence in the results. However, while they are very useful for indicating quality trends (for example system B with a BLEU of 79 can quite safely be expected to be better than old system A with a BLEU of 60), their interpretation remains cryptic to many users: “What does it mean if my English to Polish MT system gets a BLEU of 50, is this good or bad?”; “Does a GTM of 89 tell me that the translations are correct and reliable in 89% of cases?” These questions are of particular interest and relevance when the machine translations are used for very specific purposes, such as to enable critical, real-time communication, or when they are intended for human post- editing. There appears to also be a certain divide, whereby non- linguists prefer automatic scores as they are deemed mathematically more reliable and less subjective, while translators put more trust into “visual” quality checks, such as scoring or reviewing samples themselves. The metric that seems to nicely bridge the gap then would be Edit Distance (or the Levenshtein algorithm)1, in so far as it is an algorithm and outputs a score, and yet allows translators to review and understand it on an actual side-by- side comparison. It is also attractive with translation 1 See for example http://www.levenshtein.net/ buyers, who will often be overwhelmed by a plethora of seemingly black box scores. The objective of the analysis proposed in this paper is to take a deeper look at different …

Content truncated to 500 words