Bad MT Systems are Good for Quality Estimation

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 200–208 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 200 Bad MT Systems are Good for Quality Estimation Iryna Tryhubyshyn⋆ tryhubyshyn@gmail.com Aleˇs Tamchyna† ales.tamchyna@phrase.com Ondˇrej Bojar⋆ bojar@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic⋆ Phrase a.s., V´aclavsk´e n´amˇest´ı 2132/47, 110 00 Prague, Czech Republic† Abstract Quality estimation (QE) is the task of predicting quality of outputs produced by machine trans- lation (MT) systems. Currently, the highest-performing QE systems are supervised and require training on data with golden quality scores. In this paper, we investigate the impact of the quality of the underlying MT outputs on the performance of QE systems. We find that QE models trained on datasets with lower-quality translations often outperform those trained on higher-quality data. We also demonstrate that good performance can be achieved by using a mix of data from different MT systems. 1 Introduction Quality Estimation (QE) involves predicting the quality of a machine-translated text based on the original text and the machine translation (MT) output (Blatz et al., 2004; Specia et al., 2009). This can be done at the word, sentence, or document level. In this paper, we focus on sentence-level QE, where the goal is to predict a score that a human assessor would attribute to the sentence. Depending on the manual evaluation process used to gather data, we can talk about different variations of the task. These include Direct Assessment QE (Graham et al., 2015), which aims to estimate the perceived quality of transla- tion, Post-editing QE, which measures the effort required to edit the translation, and MQM QE (Lommel et al., 2014), which identifies critical errors in the translation. Evaluating a QE system means checking how closely its predictions match manual scores on a held-out set. QE systems are closely tied to MT systems in many ways. Their performance can vary greatly depending on the MT system on which they are being evaluated. The current high-performing solutions for quality estimation are based on supervised methods, which in turn makes these QE systems dependent on the specifics of the MT systems used to create the training data. It is not clear which MT system should be used to create a QE system with the best performance. The contributions of this experimental work are as follows: 1. We examine the relationship between MT system quality and QE system performance by training QE models on datasets that consist of the same source data but different transla- tions produced by MT of varying quality. 2. We evaluate the models on evaluation datasets from different domains and show that the QE system trained on translations from low-quality MT systems outperforms the QE sys- tem trained on translations from high-quality MT systems. 201 3. We demonstrate that QE systems trained on a mix of translations from different MT models also show …

Content truncated to 500 words