The MUCOW word sense disambiguation test suite at WMT 2020

Proceedings of the 5th Conference on Machine Translation (WMT), pages 365–370 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 365 The MUCOW word sense disambiguation test suite at WMT 2020 Yves Scherrer Alessandro Raganato University of Helsinki {name.surname}@helsinki.fi J¨org Tiedemann Abstract This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT sys- tems to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. MUCOW is created au- tomatically using existing resources, and the evaluation process is also entirely automated. We evaluate all participating systems of the language pairs English → Czech, English ↔ German, and English → Russian and compare the results with those obtained at WMT 2019. While current NMT systems are fairly good at handling ambiguous source words, we could not identify any substantial progress – at least to the extent that it is measurable by the MU- COW method – in that area over the last year. 1 Introduction At WMT 2019, we introduced the MUCOW (mul- tilingual contrastive word sense disambiguation) test suite (Raganato et al., 2019) and evaluated the news task submissions of nine translation di- rections with it.1 We observed that systems gen- erally performed quite well on word sense dis- ambiguation, but found a big gap between in- domain and out-of-domain disambiguation perfor- mance for some translation directions, in particu- lar with constrained systems. For WMT 2020, we reuse the same test suite for the same language pairs. This gives us the opportunity to measure the advancement of ma- chine translation within a year. We expect the larger training data sets and the model improve- ments to have a small but positive impact on trans- lation quality in general, and word sense disam- biguation performance in particular. 1The MUCOW test suite is available at http:// github.com/Helsinki-NLP/MuCoW. 2 The MUCOW test suite MUCOW (Raganato et al., 2019) is a language- independent method for automatically building test suites to assess the capabilities of MT systems to disambiguate between ambiguous words in the source language. The version of MUCOW used for WMT 2019 involves the following steps: 1. Identify ambiguous source nouns and their translations, using word-aligned and tagged parallel corpora from the OPUS collection (Tiedemann, 2012). 2. Cluster the translations into senses. First, we query BabelNet (Navigli and Ponzetto, 2012), a wide-coverage multilingual encyclo- pedic dictionary, to assign senses (synsets) to words. Second, we refine the results with the SW2V sense embeddings (Mancini et al., 2017). 3. Select sentences with ambiguous words and assign them sets of correct and incorrect tar- get translations. We evaluated the systems participating in the WMT 2019 news translation task with MUCOW for the language pairs English → Czech, English ↔ German, English ↔ Finnish, English ↔ Rus- sian, and English ↔ Lithuanian. A substantial amount of MUCOW sentences and senses come from the OpenSubtitles2018 cor- pus, but most systems participating at WMT …

Content truncated to 500 words