PreQuEL: Quality Estimation of Machine Translation Outputs in Advance

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11170 - 11183 December 7-11, 2022 ©2022 Association for Computational Linguistics PreQuEL: Quality Estimation of Machine Translation Outputs in Advance Shachar Don-Yehiya Leshem Choshen Omri Abend School of Computer Science and Engineering, The Hebrew University of Jerusalem {first.last}@mail.huji.ac.il Abstract We present the task of PreQuEL, Pre-(Quality- Estimation) Learning. A PreQuEL system pre- dicts how well a given sentence will be trans- lated, without recourse to the actual transla- tion, thus eschewing unnecessary resource al- location when translation quality is bound to be low. PreQuEL can be defined relative to a given MT system (e.g., some industry ser- vice) or generally relative to the state-of-the- art. From a theoretical perspective, PreQuEL places the focus on the source text, tracing properties, possibly linguistic features, that make a sentence harder to machine translate. We develop a baseline model for the task and analyze its performance. We also de- velop a data augmentation method (from par- allel corpora), that improves results substan- tially. We show that this augmentation method can improve the performance of the Quality- Estimation task as well.1 We investigate the properties of the input text that our model is sensitive to, by testing it on challenge sets and different languages. We conclude that it is aware of syntactic and semantic distinctions, and correlates and even over-emphasizes the importance of standard NLP features. 1 Introduction Can we tell if a sentence is difficult to automati- cally translate, without actually translating it? We argue that this question has important practical and theoretical value. From a practical standpoint, a PreQuEL system can save trouble and cost. For individuals or com- panies who do not maintain their in-house Machine Translation (MT) system, translating a big amount of data might turn out to be expensive. There are several automatic translation services offered, but there is no prior indication that any of them would do a good enough job. Therefore, after purchasing 1Code and data are available in: https://github. com/shachardon/PreQuEL Figure 1: Quality Estimation (QE) vs. PreQuEL. In the PreQuEL task, the score is generated from the source sentence directly, before investing the necessary resources to translate. translations, one would probably want to estimate the quality of the translations, with human annota- tors or an automatic Quality Estimation (QE) sys- tem (Blatz et al., 2004; Specia et al., 2009). After spending these resources, there is still a chance that the translation quality will be judged unsatisfying, and there will be no choice but to hire human trans- lators. The decision between the automatic route and the much more expensive alternative of hiring human translators can be informed by a PreQuEL model, before investing the resources to actually translate. See figure 1. Theoretically, PreQuEL allows insight into the performance boundaries of state-of-the-art MT methods. It is common practice to search in retro- spect what kinds of text phenomena are difficult for existing MT systems to handle (Toral and Sánchez- Cartagena, 2017). Such analysis helps in …

Content truncated to 500 words