Exploring Word Sense Disambiguation Abilities of Neural Machine Translation Systems (Non-archival Extended Abstract)

Exploring Word Sense Disambiguation Abilities of Neural Machine Translation Systems Rebecca Marvin becky@jhu.edu Philipp Koehn phi@jhu.edu Department of Computer Science, Johns Hopkins University, Baltimore, 21218, USA Abstract Neural machine translation systems have been shown to achieve state-of-the-art translation per- formance for many language pairs. In order to produce a correct translation, MT systems must learn how to disambiguate words with multiple senses and pick the correct translation. We ex- plore the extent to which the word embeddings for ambiguous words are able to disambiguate senses at deeper layers of the NMT encoder, which are thought to represent words with sur- rounding context. Consistent with previous research, we find that the NMT system fails to translate many ambiguous words correctly. We provide an evaluation framework to use for proposed improvements to word sense disambiguation abilities of NMT systems. 1 Introduction Neural machine translation systems have to be able to perform many different linguistic tasks successfully in order to obtain good translations. For example, MT systems have to be able to deal with syntactic reordering, semantic relationships, co-reference, and discourse roles, among other phenomena. The obvious question that arises is: how well are state-of-the-art NMT sys- tems doing at detecting linguistic features? This question is not new. Statistical machine translation (SMT) systems have achieved consistently high BLEU scores because they explicitly try to model features such as word or phrase alignments. For lower-resource languages, SMT systems have been shown to outperform NMT systems, but NMT systems overtake SMT once there is enough training data (Koehn and Knowles, 2017). Recent work has looked at the ability of neural systems to learn syntactic and morphological features. Specifically, Belinkov et al. (2017) showed that recurrent neural net- works are able to achieve high accuracy on tasks such as predicting morphological or part of speech tags and Linzen et al. (2016) showed that RNNs follow similar patterns as humans with respect to sentences that are grammatical or ungrammatical in agreement structure. Addition- ally, specific RNN cells can be shown to have high correlation with features such as sentence length (Karpathy et al., 2016), part of speech (Ding et al., 2017), or whether or not the RNN has finished a relative clause (Linzen et al., 2016). Another linguistic issue NMT systems have to deal with is translating words in the source language that might have multiple translations in the target language. When these words don’t differ orthographically, this task is known as word sense disambiguation. Typically, humans can successfully translate these kinds of words by looking at the contexts in which they appear. If NMT systems are able to successfully translate these words, it seems likely that they would have had to learn something about word sense disambiguation. There has been much research on improving machine translation performance by simulta- neously improving word sense disambiguation (Vickrey et al., 2005; Chan et al., 2007; Carpuat Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 125 and Wu, 2007) for SMT …

Content truncated to 500 words