Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 124–132 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64013 Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation Brian Thompson† Huda Khayrallah† Antonios Anastasopoulos‡ Arya D. McCarthy† Kevin Duh† Rebecca Marvin† Paul McNamee† Jeremy Gwinnup◦ Tim Anderson◦ and Philipp Koehn† †Johns Hopkins University, ‡University of Notre Dame, ◦Air Force Research Laboratory {brian.thompson, huda, arya, becky, mcnamee, phi}@jhu.edu, aanastas@nd.edu, kevinduh@cs.jhu.edu, {jeremy.gwinnup.1, timothy.anderson.20}@us.af.mil Abstract To better understand the effectiveness of con- tinued training, we analyze the major compo- nents of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component’s contri- bution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is sur- prisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitiv- ity analysis metric, suggesting that the out-of- domain model can provide a good generic ini- tialization for the new domain. 1 Introduction Neural Machine Translation (NMT) has supplanted Phrase-Based Machine Translation (PBMT) as the standard for high-resource machine translation. This has necessitated new domain adaptation meth- ods, because PBMT adaptation methods primarily rely on adapting the language model and phrase ta- ble using interpolation or back-off schemes (see §2). Continued training (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016), also referred to as fine-tuning, is one of the most popular methods for NMT adaptation, due to its strong performance. In contrast to the PBMT literature, little research has focused on why continued training is effective or on what happens to NMT models during con- tinued training. Motivated by domain adaptation analysis in PBMT (Haddow and Koehn, 2012; Duh et al., 2010; Irvine et al., 2013), this work proposes a simple freezing subnetworks technique and uses it to gain insight into how the various components of an NMT system behave during continued training. Wash your hands Wasch dir die Hände Softmax Decoder Source Target Embedding Embedding Encoder Figure 1: Visualization of an NMT system seg- mented into components. We segment the model into five subnetworks, which we refer to as components, denoted in Fig- ure 1: the source embeddings, encoder, decoder (which includes the attention mechanism), the soft- max (used to denote the decoder output embed- dings and biases), and the target embeddings. We freeze components one at a time during con- tinued training to see how much the adaptation depends on each component. We also experiment with freezing everything except one component to determine each component’s capacity to adapt to the new domain on its own. In order to further analyze continued training, we examine the magnitude of change in model com- ponents during continued training of the network, under both normal and freezing training conditions. We also conduct sensitivity …

Content truncated to 500 words