An Evaluation Benchmark for Testing the Word Sense Disambiguation Capabilities of Machine Translation Systems

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3668–3675 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3668 An Evaluation Benchmark for Testing the Word Sense Disambiguation Capabilities of Machine Translation Systems Alessandro Raganato, Yves Scherrer, J¨org Tiedemann University of Helsinki Department of Digital Humanities, Faculty of Arts {alessandro.raganato,yves.scherrer,jorg.tiedemann}@helsinki.fi Abstract Lexical ambiguity is one of the many challenging linguistic phenomena involved in translation, i.e., translating an ambiguous word with its correct sense. In this respect, previous work has shown that the translation quality of neural machine translation systems can be improved by explicitly modelling the senses of ambiguous words. Recently, several evaluation test sets have been proposed to measure the word sense disambiguation (WSD) capability of machine translation systems. However, to date, these evaluation test sets do not include any training data that would provide a fair setup measuring the sense distributions present within the training data itself. In this paper, we present an evaluation benchmark on WSD for machine translation for 10 language pairs, comprising training data with known sense distributions. Our approach for the construction of the benchmark builds upon the wide-coverage multilingual sense inventory of BabelNet, the multilingual neural parsing pipeline TurkuNLP, and the OPUS collection of translated texts from the web. The test suite is available at http://github.com/Helsinki-NLP/MuCoW. Keywords: machine translation, word sense disambiguation, evaluation benchmark, test suite 1. Introduction In recent years, several advances have been made in Word Sense Disambiguation (WSD) (Raganato et al., 2017; Loureiro and Jorge, 2019; Kumar et al., 2019). WSD mod- els that tackle lexical ambiguity effectively bring numerous benefits to a variety of downstream tasks and applications, such as information retrieval and extraction (Zhong and Ng, 2012; Delli Bovi et al., 2015) and text categorization (Flekova and Gurevych, 2016; Pilehvar et al., 2017; Sinoara et al., 2019; Shimura et al., 2019). Another downstream ap- plication is machine translation (MT), where word sense disambiguation plays a crucial role to select the correct translation sense for each ambiguous word (Rios et al., 2017; Pu et al., 2018; Liu et al., 2018). To measure specific linguistic phenomena in machine trans- lation, several test suites – or challenge sets – have emerged (Popovi´c and Castilho, 2019). They are evaluation bench- marks that focus on particular linguistic phenomena and provide specific evaluation criteria or metrics. For lexical ambiguity of nouns, to our knowledge, two major test suites exist: ContraWSD (Rios et al., 2017; Rios et al., 2018) and MUCOW (Raganato et al., 2019). Both test suites are available in two variants: scoring and translation. The first variant relies on the ability of neural machine translation systems to score given translations: a sentence containing an ambiguous source word is paired with the correct ref- erence translation and with a modified translation in which the ambiguous word has been replaced by a word of a dif- ferent sense. A contrast is considered successfully detected if the reference translation obtains a higher score than the artificially modified …

Content truncated to 500 words