DiBiMT: A Novel Benchmark for Measuring Word Sense Disambiguation Biases in Machine Translation

DIBIMT: A Novel Benchmark for Measuring Word Sense Disambiguation Biases in Machine Translation Niccolò Campolungo∗ Sapienza University of Rome campolungo@di.uniroma1.it Federico Martelli∗ Sapienza University of Rome martelli@di.uniroma1.it Francesco Saina SSML Carlo Bo, Rome f.saina@ssmlcarlobo.it Roberto Navigli Sapienza University of Rome navigli@diag.uniroma1.it Abstract Lexical ambiguity poses one of the greatest challenges in the field of Machine Translation. Over the last few decades, multiple efforts have been undertaken to investigate incorrect translations caused by the polysemous nature of words. Within this body of research, some studies have posited that models pick up se- mantic biases existing in the training data, thus producing translation errors. In this paper, we present DIBIMT, the first entirely manually- curated evaluation benchmark which enables an extensive study of semantic biases in Machine Translation of nominal and verbal words in five different language combinations, namely, English and one or other of the fol- lowing languages: Chinese, German, Italian, Russian and Spanish. Furthermore, we test state-of-the-art Machine Translation systems, both commercial and non-commercial ones, against our new test bed and provide a thor- ough statistical and linguistic analysis of the results. We release DIBIMT at https:// nlp.uniroma1.it/dibimt as a closed benchmark with a public leaderboard. 1 Introduction The polysemous nature of words poses a long- standing challenge in a wide range of Natural Lan- guage Processing (NLP) tasks such as Word Sense Disambiguation (Navigli, 2009; Bevilacqua et al., 2021) (WSD), Information Retrieval (Krovetz and Croft, 1992) (IR) and Machine Translation (Emelin et al., 2020) (MT). In MT, some research works have addressed the ability of systems to disambiguate polysemous words. For instance, given the sentence He poured a shot of whiskey, the polysemous target word shot unequivocally means a small quantity and therefore a possible translation into Italian could be: Versò un goccio di whiskey. However, some MT systems propose the following translation: Versò uno sparo ∗ Equal contribution. di whiskey in which the noun sparo means gun- shot. This is one of many examples that seem to encourage a deeper performance analysis in sce- narios in which MT systems are required to deal with polysemous words and, specifically, with in- frequent meanings of polysemous words. Although state-of-the-art MT systems, both commercial and non-commercial ones, achieve impressive BLEU scores on standard benchmarks, in our work we demonstrate that they still present significant limi- tations when dealing with infrequent word senses, which standard metrics fail to recognize. In the last few decades, attempts have been made to investigate the aforementioned phenomena. In fact, recent studies have observed a direct correla- tion between semantic biases in the training data and semantic errors in translation. However, their findings are limited by the following shortcomings: i) they are not based on entirely manually-curated benchmarks; ii) they rely heavily on automatically- generated resources to determine the correctness of a translation; and iii) they do not cover multiple language combinations. In this work, we address the aforementioned drawbacks and present DIBIMT, to the best of our knowledge the first fully manually-curated evalua- tion benchmark aimed …

Content truncated to 500 words