MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 569 - 577 December 7-8, 2022 ©2022 Association for Computational Linguistics MATESE: Machine Translation Evaluation as a Sequence Tagging Problem Stefano Perrella1, Lorenzo Proietti1, Alessandro Scirè1,2 Niccolò Campolungo1 and Roberto Navigli1 1Sapienza NLP Group, Sapienza University of Rome 2Babelscape, Italy {stefano.perrella, l.proietti, alessandro.scire}@uniroma1.it campolungo@di.uniroma1.it navigli@diag.uniroma1.it Abstract Starting from last year, WMT human evalu- ation has been performed within the Multi- dimensional Quality Metrics (MQM) frame- work, where human annotators are asked to identify error spans in translations, alongside an error category and a severity. In this pa- per, we describe our submission to the WMT 2022 Metrics Shared Task, where we propose using the same paradigm for automatic evalua- tion: we present the MATESE metrics, which reframe machine translation evaluation as a sequence tagging problem. Our submission also includes a reference-free metric, denom- inated MATESE-QE. Despite the paucity of the openly available MQM data, our metrics obtain promising results, showing high levels of correlation with human judgements, while also enabling an evaluation that is interpretable. Moreover, MATESE-QE can also be employed in settings where it is infeasible to curate refer- ence translations manually. 1 Introduction and Related Work For many years, Machine Translation (MT) has mainly been evaluated using untrained evaluation techniques, such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and CHRF (Popovi´c, 2015), which rely heavily on lexical-level matching of either token, or character, n-grams. Unfortunately, these metrics present two major drawbacks: i) it is not possible to carry out the eval- uation without manually-curated references and, most importantly, ii) the evaluation is too depen- dent on the surface form of the translation, and its reference. More recently, attempts have been made to address these problems using machine-learned metrics, which have shown better correlations with human judgements (Mathur et al., 2020). More specifically, last year’s WMT Metrics Shared Task saw C-SPECPN (Takahashi et al., 2021), BLEURT- 201 and COMET-MQM_2021 (Rei et al., 2021) emerge as distinctly better than the other partici- pants (Freitag et al., 2021b). These metrics con- sist of regression models trained to mimic hu- man annotators by directly assigning quality scalar scores to candidate translations. In detail, COMET- MQM_2021 is based on the Estimator architecture introduced by Rei et al. (2020), where features ex- tracted from the embeddings of the source sentence, candidate translation, and reference translation are passed to a feed-forward regressor; C-SPEC first concatenates the embeddings derived from paired inputs of candidate-source and candidate-reference, and then passes the resulting vector to a multi-layer perceptron; BLEURT, instead, feeds the candidate translation and its reference to Rebalanced mBERT (Chung et al., 2021), and regresses on the rep- resentation provided by the [CLS] token. More- over, BLEURT and C-SPEC add automatically- generated negative pairs to the standard training data: BLEURT applies random token perturbations, while C-SPEC uses Word Attribute Transfer to re- place words in the translations. Although undoubt- edly effective, regression metrics have the major drawback of not being interpretable, meaning that …

Content truncated to 500 words