Robust MT Evaluation with Sentence-level Multilingual Augmentation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 469 - 478 December 7-8, 2022 ©2022 Association for Computational Linguistics Robust MT evaluation with Sentence-level Multilingual Augmentation Duarte M. Alves∗1, Ricardo Rei1,3,4, Ana C. Farinha3, José G. C. de Souza3, André F. T. Martins1,2,3 1Instituto Superior Técnico, University of Lisbon, Portugal 2Instituto de Telecomunicações, Lisbon, Portugal 3Unbabel, Lisbon, Portugal, 4INESC-ID, Lisbon, Portugal Abstract Automatic translations with critical errors may lead to misinterpretations and pose several risks for the user. As such, it is important that Ma- chine Translation Evaluation systems are robust to these errors in order to increase the reliabil- ity and safety of the translation process. Here we introduce SMAUG, a novel Sentence-level Multilingual AUGmentation approach for gen- erating translations with critical errors and ap- ply this approach to create a test set to evaluate the robustness of Machine Translation metrics to these errors. We show that current State-of- the-Art methods are improving their capability to distinguish translations with and without crit- ical errors and to penalize the first accordingly. We also show that metrics tend to struggle with errors related to named entities and numbers and that there is a high variance in the robust- ness of current methods to translations with critical errors. 1 Introduction In recent years, Machine Translation (MT) systems have been used in diverse real world environments. However, widespread adoption of these systems raises many concerns, namely in the quality of their outputs. Ideally, human translators would evaluate generated translations but this process is expensive and slow. As an alternative, automatic Machine Translation Evaluation relies on external systems to measure the quality of generated translations. As a crucial aspect of Machine Translation Eval- uation, it is vital to ensure that generated sentences do not contain critical errors. As detailed in Specia et al. (2021), translations with such errors devi- ate in meaning from their source sentence in ways that may lead to misinterpretations and pose health, safety, legal, reputation, religious or financial im- plications. Specia et al. (2021) group these trans- lations into three categories, based on how their ∗Corresponding author: duartemalves@tecnico.ulisboa.pt meaning deviates from the source sentence. Mis- translation errors have critical content in the source sentence translated into a different meaning, not translated (the content remains in the source lan- guage), or translated into gibberish. Hallucination errors introduce content in the translated sentence that is not present in the source sentence. Deletion errors exclude important content from the source sentence. In this work, we propose SMAUG1, a Sentence- level Multilingual AUGmentation framework to generate translations with critical errors, targeting all the aforementioned critical error categories. We also introduce a novel test set to analyse the robustness of MT Evaluation systems to criti- cal errors. This test set was created with the pro- posed augmentation framework and submitted to the WMT22 Challenge Sets Sub-task (Freitag et al., 2022). Finally, we present the results obtained from evaluating metrics submitted to the WMT22 Met- rics Shared Task with the developed test set. We …

Content truncated to 500 words