Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 2062–2068 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2062 Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation Brian Thompson† Jeremy Gwinnup◦ Huda Khayrallah† Kevin Duh† Philipp Koehn† †Johns Hopkins University, ◦Air Force Research Laboratory {brian.thompson, huda, phi}@jhu.edu, kevinduh@cs.jhu.edu, jeremy.gwinnup.1@us.af.mil Abstract Continued training is an effective method for domain adaptation in neural machine transla- tion. However, in-domain gains from adapta- tion come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catas- trophic forgetting of general-domain knowl- edge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without for- getting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading in- domain performance, outperforming the pre- vious state-of-the-art. We also explore the full range of general-domain performance avail- able when some in-domain degradation is ac- ceptable. 1 Introduction Neural Machine Translation (NMT) performs poorly without large training corpora (Koehn and Knowles, 2017). Domain adaptation is required when there is sufficient data in the desired lan- guage pair but insufficient data in the desired do- main (the topic, genre, style or level of formality). This work focuses on the supervised domain adap- tation problem where a small in-domain parallel corpus is available for training. Continued train- ing (Luong and Manning, 2015; Sennrich et al., 2015) (also called fine-tuning), where a model is first trained on general-domain data and then do- main adapted by training on in-domain data, is a popular approach in this setting as it leads to em- pirical improvements in the targeted domain. One downside of continued training is that the adapted model’s ability to translate general- domain sentences is severely degraded during adaptation (Freitag and Al-Onaizan, 2016). We in- terpret this drop in general-domain performance as catastrophic forgetting (Goodfellow et al., 2013) of general-domain translation knowledge. Degra- dation of general-domain performance may be problematic when the domain adapted NMT sys- tem is used to translate text outside its target do- main, which can happen if there is a mismatch be- tween the data available for domain-specific train- ing and the test data. Poor performance may also concern end users of these MT systems who are expecting good performance on ‘easy’ generic sentences.1 Elastic Weight Consolidation (EWC) (Kirk- patrick et al., 2017) is a method for training neu- ral networks to learn a new task without for- getting previously learned tasks. We extend EWC to continued training in NMT (see §3): Our first task is to translate general-domain sen- tences, and our second is to translate domain- specific sentences (without forgetting how to translate general-domain sentences). EWC works by adding a per-parameter regularizer, based on the Fisher Information matrix, while training on the second task. At a high level, the regulariza- tion term keeps parameters which are important to general-domain performance close to the ini- tial general-domain model values during contin- ued training, while allowing …

Content truncated to 500 words