Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10173–10185 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10173 Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output Raksha Shenoy1,2, Nico Herbig1, Antonio Krüger1, Josef van Genabith1,2 1German Research Center for Artificial Intelligence (DFKI) Saarland Informatics Campus, Germany 2Department of Language Science and Technology Saarland University, Germany rakshamanel08shenoy@gmail.com {nico.herbig, krueger, josef.van_genabith}@dfki.de Abstract Compared to fully manual translation, post- editing (PE) machine translation (MT) output can save time and reduce errors. Automatic word-level quality estimation (QE) aims to pre- dict the correctness of words in MT output and holds great promise to aid PE by flag- ging problematic output. Quality of QE is crucial, as incorrect QE might lead to transla- tors missing errors or wasting time on already correct MT output. Achieving accurate auto- matic word-level QE is very hard, and it is cur- rently not known (i) at what quality threshold QE is actually beginning to be useful for hu- man PE, and (ii), how to best present word- level QE information to translators. In partic- ular, should word-level QE visualization indi- cate uncertainty of the QE model or not? In this paper, we address both research questions with real and simulated word-level QE, visual- izations, and user studies, where time, subjec- tive ratings, and quality of the final translations are assessed. Results show that current word- level QE models are not yet good enough to support PE. Instead, quality levels of ≥ 80% F1 are required. For helpful quality levels, a vi- sualization reflecting the uncertainty of the QE model is preferred. Our analysis further shows that speed gains achieved through QE are not merely a result of blindly trusting the QE sys- tem, but that the quality of the final translations also improves. The threshold results from the paper establish a quality goal for future word- level QE research. 1 Introduction Advances in Machine Translation (MT) have made MT a key component in many professional transla- tion workflows, where human post-editors identify and correct mistakes in raw MT output. Overall, Post-Editing (PE) saves time and reduces errors (Moorkens and Brien, 2017; Lagoudaki, 2009; Ya- mada, 2014; Green et al., 2013), however, for sen- tences of low MT quality, PE can take longer than manually translating from scratch (Specia et al., 2018). In order to better support and guide post- editors, it would therefore be helpful to indicate MT quality and flag potential errors. Quality Estimation (QE) focuses on this goal by predicting how good MT output is based on factors such as fluency, comprehensibility, and adequacy (Moorkens et al., 2018). In contrast to standard MT evaluation methods such as BLEU (Papineni et al., 2002) and TER (Snover et al., 2006), QE does not rely on reference translations for compar- ison. Instead, QE can be framed as a supervised machine learning task, predicting the quality of an MT output for a source sentence (Turchi et al., 2013). QE models can be …

Content truncated to 500 words