Simple, Scalable Adaptation for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1538–1548, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1538 Simple, Scalable Adaptation for Neural Machine Translation Ankur Bapna Google AIOrhan Firat {ankurbpn,orhanf}@google.com Abstract Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multi- lingual NMT. Experiments on domain adapta- tion demonstrate that our proposed approach is on par with full fine-tuning on various do- mains, dataset sizes and model capacities. On a massively multilingual dataset of 103 lan- guages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards uni- versal machine translation. 1 Introduction Recent developments in deep learning have led to significantly improved quality on Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017). While NMT per- formance on sentence level translation for high resource languages seems to be dramatically im- proved (Wu et al., 2016; Hassan et al., 2018), per- formance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013; Koehn and Knowles, 2017; Farajian et al., 2017; Dong et al., 2015; Zoph et al., 2016). This has generated significant interest in adaptation ap- proaches, that can leverage the huge amounts of parallel data available for high resource languages, to improve translation performance on low re- source tasks. In this work we focus on two adap- tation tasks: (i) Domain Adaptation, to improve the performance of in-domain translation by lever- aging out-of-domain datasets (Luong and Man- ning, 2015; Freitag and Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al., 2015; Firat et al., 2016; Johnson et al., 2016; Zoph et al., 2016; Neubig and Hu, 2018; Aharoni et al., 2019; Ari- vazhagan et al., 2019). While several approaches have been explored in literature (Chu and Wang, 2018), full fine- tuning of model parameters remains the dominant approach for adapting to new domains and lan- guages (Luong and Manning, 2015; Neubig and Hu, 2018). However, fine-tuning requires training and maintaining a separate model for every lan- guage, for every domain. As the number of lan- guages and domains grow, training, maintaining and serving a separate model for every task be- comes infeasible. This is further exacerbated by the increasing model capacity of state-of-the-art NMT models …

Content truncated to 500 words