Non-Parametric Adaptation for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1921–1931 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1921 Non-Parametric Adaptation for Neural Machine Translation Ankur Bapna Google AI ankurbpn@google.com Orhan Firat Google AI orhanf@google.com Abstract Neural Networks trained with gradient descent are known to be susceptible to catastrophic forgetting caused by parameter shift during the training process. In the context of Neural Ma- chine Translation (NMT) this results in poor performance on heterogeneous datasets and on sub-tasks like rare phrase translation. On the other hand, non-parametric approaches are im- mune to forgetting, perfectly complementing the generalization ability of NMT. However, attempts to combine non-parametric or re- trieval based approaches with NMT have only been successful on narrow domains, possibly due to over-reliance on sentence level retrieval. We propose a novel n-gram level retrieval ap- proach that relies on local phrase level simi- larities, allowing us to retrieve neighbors that are useful for translation even when overall sentence similarity is low. We complement this with an expressive neural network, al- lowing our model to extract information from the noisy retrieved context. We evaluate our semi-parametric NMT approach on a hetero- geneous dataset composed of WMT, IWSLT, JRC-Acquis and OpenSubtitles, and demon- strate gains on all 4 evaluation sets. The semi- parametric nature of our approach opens the door for non-parametric domain adaptation, demonstrating strong inference-time adapta- tion performance on new domains without the need for any parameter updates. 1 Introduction Over the last few years, neural sequence to se- quence models (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014) have revolution- ized the field of machine translation by signif- icantly improving translation quality over their phrase based counterparts (Sennrich et al., 2015; Wu et al., 2016; Zhou et al., 2016). With more gains arising from continued research on new neu- ral network architectures and accompanying train- ing techniques (Vaswani et al., 2017; Gehring et al., 2017; Chen et al., 2018), NMT researchers, both in industry and academia, have doubled down on their ability to train high capacity models on large corpora with gradient based optimization. However, despite huge improvements in overall translation quality NMT has shown some glaring weaknesses, including idiom processing, and rare word or phrase translation (Koehn and Knowles, 2017; Isabelle et al., 2017; Lee et al., 2018) - tasks that should be easy if the model could re- tain learned information from individual training examples. NMT has also been shown to per- form poorly when dealing with multi-domain data (Farajian et al., 2017a). This ‘catastrophic for- getting’ problem has been well-studied in tradi- tional neural network literature, caused by param- eter shift during the training process (McCloskey and Cohen, 1989; Santoro et al., 2016). Non- parametric methods, on the other hand, are re- sistant to forgetting but are prone to over-fitting due to their reliance on individual training exam- ples. We focus on a non-parametric extension to NMT, hoping to combine the generalization abil- ity of neural networks with the eidetic memory …

Content truncated to 500 words