Improving Lexical Choice in Neural Machine Translation

Proceedings of NAACL-HLT 2018, pages 334–343 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Improving Lexical Choice in Neural Machine Translation Toan Q. Nguyen and David Chiang Department of Computer Science and Engineeering University of Notre Dame {tnguye28,dchiang}@nd.edu Abstract We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner prod- uct of a vector representing the context with all possible output word embeddings, rewards fre- quent words disproportionately, and we pro- pose to fix the norms of both vectors to a con- stant value. Second, we integrate a simple lex- ical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improve- ments of up to +4.3 BLEU, surpassing phrase- based translation in nearly all settings.1 1 Introduction Neural network approaches to machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Lu- ong et al., 2015a; Gehring et al., 2017) are appeal- ing for their single-model, end-to-end training pro- cess, and have demonstrated competitive perfor- mance compared to earlier statistical approaches (Koehn et al., 2007; Junczys-Dowmunt et al., 2016). However, there are still many open prob- lems in NMT (Koehn and Knowles, 2017). One particular issue is mistranslation of rare words. For example, consider the Uzbek sentence: Source: Ammo muammolar hali ko’p, deydi amerikalik olim Entoni Fauchi. Reference: But still there are many problems, says American scientist Anthony Fauci. Baseline NMT: But there is still a lot of problems, says James Chan. At the position where the output should be Fauci, the NMT model’s top three candidates are Chan, 1The code for this work can be found at https://github.com/tnq177/improving_lexical_ choice_in_nmt Fauci, and Jenner. All three surnames occur in the training data with reference to immunologists: Fauci is the director of the National Institute of Allergy and Infectious Diseases, Margaret (not James) Chan is the former director of the World Health Organization, and Edward Jenner invented smallpox vaccine. But Chan is more frequent in the training data than Fauci, and James is more frequent than either Anthony or Margaret. Because NMT learns word representations in continuous space, it tends to translate words that “seem natural in the context, but do not reflect the content of the source sentence” (Arthur et al., 2016). This coincides with other observations that NMT’s translations are often fluent but lack accu- racy (Wang et al., 2017b; Wu et al., 2016). Why does this happen? At each time step, the model’s distribution over output words e is p(e) ∝ exp � We · ˜h + be � where We and be are a vector and a scalar depend- ing only on e, and ˜h is a vector depending only on the source sentence and previous output words. We propose two modifications to this layer. First, we argue that the term We · ˜h, which …

Content truncated to 500 words