HPLT: High Performance Language Technologies

BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation Taisiya Glushkova1,3 Chrysoula Zerva1,3 Andr´e F. T. Martins1,2,3 1Instituto de Telecomunicac¸˜oes 2Unbabel 3Instituto Superior T´ecnico & LUMLIS (Lisbon ELLIS Unit) {taisiya.glushkova, chrysoula.zerva, andre.t.martins}@tecnico.ulisboa.pt Abstract Although neural-based machine transla- tion evaluation metrics, such as COMET or BLEURT, have achieved strong corre- lations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered as crit- ical errors, such as deviations in entities and numbers. In contrast, traditional eval- uation metrics such as BLEU or CHRF, which measure lexical or character overlap between translation hypotheses and human references, have lower correlations with hu- man judgements but are sensitive to such deviations. In this paper, we investigate sev- eral ways of combining the two approaches in order to increase robustness of state-of- the-art evaluation methods to translations with critical errors. We show that by us- ing additional information during training, such as sentence-level features and word- level tags, the trained metrics improve their capability to penalize translations with spe- cific troublesome phenomena, which leads to gains in correlation with human judg- ments and on recent challenge sets on sev- eral language pairs.1 1 Introduction Trainable machine translation (MT) evaluation models, such as COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020), generally achieve higher correlations with human judgments, thanks © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 1Our code and data are available at: https://github.com/ deep-spin/robust MT evaluation to leveraging pretrained language models. How- ever, they often fail at detecting certain types of errors and deviations from the source, for exam- ple related to translations of numbers and entities (Amrhein and Sennrich, 2022). As a result, their quality predictions are sometimes hard to interpret and not always trustworthy. In contrast, traditional lexical-based metrics, such as BLEU (Papineni et al., 2002) or CHRF (Popovi´c, 2015)—despite their many limitations—are considerably more sensitive to these errors, due to their nature, and are also more interpretable, since the scores can be traced back to the character or n-gram overlap. This paper investigates and compares methods that combine the strengths of neural-based and lex- ical approaches, both at the sentence level and at the word level. This is motivated by the findings of previous works, which demonstrate in detail that the COMET MT evaluation metric struggles to han- dle errors like deviation in numbers, wrong named entities in generated translations, deletions that ex- clude important content from the source sentence, insertions of extra words that are not present in the source sentences, and a few others (Amrhein and Sennrich, 2022; Alves et al., 2022). While data augmentation techniques alleviate the problem to some extent (Alves et al., 2022), the gains seem to be relatively modest. In this paper we investigate alternative methods that take advantage of lexical information and go beyond the use of various aug- mentation techniques and synthetic data. We focus on increasing robustness of …

Content truncated to 500 words