ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics Chantal Amrhein1∗and Nikita Moghe2∗and Liane Guillou2∗ 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh amrhein@cl.uzh.ch, nikita.moghe@ed.ac.uk, lguillou@ed.ac.uk Abstract As machine translation (MT) metrics improve their correlation with human judgement ev- ery year, it is crucial to understand the limi- tations of such metrics at the segment level. Specifically, it is important to investigate met- ric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES1, a Translation Accuracy ChallengE Set, consisting of 68 phe- nomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) com- bining metrics with different strengths, b) de- veloping metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings. 1 Introduction Challenge sets have already been created for mea- suring the success of systems or metrics on a partic- ular phenomenon of interest for a range of NLP tasks, including but not limited to: Sentiment Analysis2 (Li et al., 2017; Mahler et al., 2017; Stali¯unait˙e and Bonfil, 2017), Natural Language Inference (McCoy and Linzen, 2019; Rocchietti et al., 2021), Question Answering (Ravichander et al., 2021), Machine Reading Comprehension (Khashabi et al., 2018), Machine Translation (MT) ∗Equal contribution by all authors. 1Our dataset is available at https://huggingface.co/ datasets/nikitam/ACES and the corresponding evaluation scripts at https://github.com/EdinburghNLP/ACES 2Submitted to the EMNLP 2017 “Build It Break It” shared task on sentiment analysis (King and Falkedal, 1990; Isabelle et al., 2017), and the more specific task of pronoun translation in MT (Guillou and Hardmeier, 2016). They are useful to compare the performance of different systems, or to identify performance improvement/degradation between a modified system and a previous iteration. In this work, we describe the University of Zurich - University of Edinburgh submission to the Challenge Sets subtask of the Conference on Machine Translation (WMT) 2022 Metrics shared task. Our Translation Accuracy ChallengE Set (ACES) consists of 36,476 examples covering 146 language pairs and representing challenges from 68 phenomena (see Appendix A.4 for the distri- bution of examples across language pairs and Ap- pendix A.5 for the distribution of language pairs across phenomena). We focus on translation accu- racy errors and base the phenomena covered in our challenge set on the Multidimensional Quality Met- rics (MQM) ontology (Lommel et al., 2014). We include phenomena ranging from simple perturba- tions involving the omission/addition of characters or tokens, to more complex examples involving mistranslation e.g. ambiguity and hallucinations in translation, untranslated elements of a sentence, discourse-level phenomena, and real-world knowl- edge. We evaluate the metrics submitted to the WMT 2022 metrics shared task …

Content truncated to 500 words