Machine Translation Quality and Post-Editor Productivity

Machine Translation Quality and Post-Editor Productivity Marina Sanchez-Torron msnc017@aucklanduni.ac.nz School of Cultures, Languages and Linguistics University of Auckland Auckland 1010, New Zealand Philipp Koehn phi@jhu.edu Department of Computer Science Johns Hopkins University Baltimore, MD 21218-2608, USA Abstract We assessed how different machine translation (MT) systems affect the post-editing (PE) pro- cess and product of professional English–Spanish translators. Our model found that for each 1-point increase in BLEU, there is a PE time decrease of 0.16 seconds per word, about 3-4%. The MT system with the lowest BLEU score produced the output that was post-edited to the lowest quality and with the highest PE effort, measured both in HTER and actual PE operations. 1 Introduction and Related Work There is a relatively fair amount of empirical research on post-editing machine translation output (PE) focusing on assessing potential time and quality improvements over human translation with or without translation memories (TM). A common finding is that, despite differences among participants, PE is on average faster than unassisted or TM-assisted translation. Some exam- ples of studies finding such speed benefits are those by Guerberof (2009), Flournoy and Duran (2009), Groves and Schmidtke (2009), Plitt and Masselot (2010) and Skadin¸ˇs et al. (2011). In terms of PE quality, studies have shown, through the use of human judgments, that PE leads to quality comparable to (Garc´ıa, 2010) or better than other types of translation (Guer- berof, 2009; Fiederer and O’Brien, 2009; Carl et al., 2011; Green et al., 2013; L¨aubli et al., 2013). There is therefore strong empirical evidence pointing at the speed and quality benefits of PE. While many factors may affect productivity in a PE workflow, MT quality is one that is relatively easy to measure. Previous studies to have investigated how MT quality affects PE speed are those by Tatsumi (2009) and O’Brien (2011). They found different levels of correlations between MT quality, measured on a sentence level with automatic metrics, and PE speed. Krings (2001) and De Sutter (2012) too found that human judgments of MT quality correlated to PE speed. Koehn and Germann (2014) found their worst MT system entailed 20% more editing activity than their best one. To the best of our knowledge, this is the first study to attempt to assess how different MT systems, of the same type but with different quality levels, affect the PE productivity of pro- fessional translators. By investigating how MT quality affects both PE time and PE quality, we aim at providing MT users and researchers with another approach to examining the usefulness of MT for PE purposes. ������������������������������������������������������ ���������������������������������������� 2 Study Setup Nine translators were hired through ProZ1, self-described as the biggest online translation work- place. Selected participants were offered a fixed compensation based on the standard, general, English–Spanish translation rates displayed on the website. Each translator post-edited four news texts of about 650 words each. Texts had similar complexity levels and were presented to translators in randomized order to dilute possible familiarization or fatigue effects. All four texts were translated …

Content truncated to 500 words