Regressive Ensemble for Machine Translation Quality Evaluation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1041–1048 November 10–11, 2021. ©2021 Association for Computational Linguistics 1041 Regressive Ensemble for Machine Translation Quality Evaluation Michal Štefánik and Vít Novotný and Petr Sojka Masaryk University, Faculty of Informatics, MIR group https://mir.fi.muni.cz {stefanik.m,witiko}@mail.muni.cz, sojka@fi.muni.cz Abstract This work introduces a simple regressive en- semble for evaluating machine translation qual- ity based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop. In both monolingual and zero-shot cross-lingual set- tings, we show a significant performance im- provement over single metrics. In the cross- lingual settings, we also demonstrate that an en- semble approach is well-applicable to unseen languages. Furthermore, we identify a strong reference-free baseline that consistently out- performs the commonly-used BLEU and ME- TEOR measures and significantly improves our ensemble’s performance. 1 Introduction Automated evaluation of text generation is challeng- ing due to many orthogonal qualitative aspects that the user expects from a text generation system. In machine translation, we can observe errors of the so-called critical category (Wulczyn et al., 2017) such as hallucinating (Lee et al., 2019), omitting parts of the input from translation, or the negation of meaning (Matusov, 2019). Consider an example of the last category: • Reference: “I never wrote this article, I just edited it.” • Hypothesis 1: “It is not my article, I just edited it.” • Hypothesis 2: “I never wrote this article, I never edited it.” In this example, all BERTScore (Zhang et al., 2019), BLEUrt (Sellam et al., 2020), and Prism (Thomp- son and Post, 2020b) metrics rank Hypothesis 2 higher than Hypothesis 1. In BLEUrt and Prism, this can be due to a known vulnerability of Trans- formers, which rely on a lexical intersection (Mc- Coy et al., 2019) if such a heuristic fits the prob- lem sufficiently well. Trivially, just counting the negations can easily remedy this specific problem. However, such a heuristic would fail in many other cases, such as when we adjust Hypothesis 1 to “It is an article of somebody else, I just edited it.”. Such cases motivate our ensemble approach that aims to expose both surface and deeper semantic properties of texts and subsequently learn to utilize these for the specific task of translation evaluation. Even though the objective might constrain the par- ticular metrics, data set, or systematically fail in some cases, another metric or their combination in the ensemble allows the flaw to be corrected. 2 Metrics for machine translation evaluation This section reviews the related work, focusing on the metrics we used in our ensemble. The standard and still widely-used surface-level metrics for the evaluation of machine translation quality are BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and TER (Snover et al., 2006). Surface- level metrics are not able to capture the proximity of meaning in cases where one text paraphrases the other, which is an ability commonly observed in deep neural language models (Lewis …

Content truncated to 500 words