Improving Machine Translation of Rare and Unseen Word Senses

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 614–624 November 10–11, 2021. ©2021 Association for Computational Linguistics 614 Improving Machine Translation of Rare and Unseen Word Senses 1Viktor Hangya, 2Qianchu Liu, 1Dario Stojanovski, 1Alexander Fraser and 2Anna Korhonen 1Center for Information and Language Processing, LMU Munich {hangyav,stojanovski,fraser}@cis.lmu.de 2Language Technology Lab, TAL, University of Cambridge, UK {ql261,alk23}@cam.ac.uk Abstract The performance of NMT systems has im- proved drastically in the past few years but the translation of multi-sense words still poses a challenge. Since word senses are not repre- sented uniformly in the parallel corpora used for training, there is an excessive use of the most frequent sense in MT output. In this work, we propose CMBT (Contextually- mined Back-Translation), an approach for im- proving multi-sense word translation leverag- ing pre-trained cross-lingual contextual word representations (CCWRs). Because of their contextual sensitivity and their large pre- training data, CCWRs can easily capture word senses that are missing or very rare in parallel corpora used to train MT. Specifically, CMBT applies bilingual lexicon induction on CCWRs to mine sense-specific target sentences from a monolingual dataset, and then back-translates these sentences to generate a pseudo paral- lel corpus as additional training data for an MT system. We test the translation quality of ambiguous words on the MuCoW test suite, which was built to test the word sense disam- biguation effectiveness of MT systems. We show that our system improves on the trans- lation of difficult unseen and low frequency word senses. 1 Introduction Recent NMT systems have remarkable perfor- mance for many languages (Vaswani et al., 2017; Xia et al., 2019) but there are still numerous areas for improvement. One such important area con- cerns the disambiguation and translation of multi- sense words. It is particularly challenging to MT systems as sense distribution is skewed with some senses rarely seen or missing in the parallel corpora. This results in the MT system producing transla- tion errors for these rare/unseen senses, causing incomprehensible output sentences. In this work we aim at improving the transla- tion of rare and unseen senses of ambiguous words. Figure 1: The pipeline of CMBT. Note that each step of the pipeline is run on the full corpus (source side of MuCoW test set). Here we just show the procedure on a single sentence as an illustration. Previously Tang et al. (2018, 2020) showed that encoder-decoder based NMT systems integrate in- formation relevant for WSD into the encoder hid- den states, but finding the correct sense is still a challenging task and NMT systems are biased toward the most frequent senses of words (Liu et al., 2018). Disambiguation errors are often due to over-reliance on training data artifacts, such as frequent word co-occurrences (e.g. hot spring is always translated as the thermal activity and not as a season), instead of a deeper understanding of the multi-sense words given the input sentences (Emelin et al., 2020). Additionally, MT systems tend to learn and use frequent words more often and disregard less …

Content truncated to 500 words