Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4269–4279 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4269 Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation Shoetsu Sato Jin Sakuma The University of Tokyo Naoki Yoshinaga Masashi Toyoda Institute of Industrial Science, the University of Tokyo Masaru Kitsuregawa Institute of Industrial Science, the University of Tokyo National Institute of Informatics {shoetsu,jsakuma,ynaga,toyoda,kitsure}@tkl.iis.u-tokyo.ac.jp Abstract Neural network methods exhibit strong perfor- mance only in a few resource-rich domains. Practitioners therefore employ domain adapta- tion from resource-rich domains that are, in most cases, distant from the target domain. Do- main adaptation between distant domains (e.g., movie subtitles and research papers), however, cannot be performed effectively due to mis- matches in vocabulary; it will encounter many domain-specific words (e.g., “angstrom”) and words whose meanings shift across domains (e.g., “conductor”). In this study, aiming to solve these vocabulary mismatches in do- main adaptation for neural machine transla- tion (NMT), we propose vocabulary adapta- tion, a simple method for effective fine-tuning that adapts embedding layers in a given pre- trained NMT model to the target domain. Prior to fine-tuning, our method replaces the em- bedding layers of the NMT model by project- ing general word embeddings induced from monolingual data in a target domain onto a source-domain embedding space. Experimen- tal results indicate that our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En→Ja and De→En translation, respectively. 1 Introduction The performance of neural machine translation (NMT) models remarkably drops in domains dif- ferent from the training data (Koehn and Knowles, 2017). Since a massive amount of parallel data is available only in a limited number of domains, do- main adaptation is often required to employ NMT in practical applications. Researchers have therefore developed fine-tuning, a dominant approach for this problem (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016; Chu et al., 2017; Thompson et al., 2018; Khayrallah et al., 2018; Bapna and Firat, 2019) (§ 2). Assuming a massive amount of Encoder-Decoder Source-domain embeddings Target-domain embeddings 2) Cross-domain emb. projection Target-domain monolingual data 1) CBoW training Small target-domain parallel data Massive source-domain parallel data 0) Pre-training 3) Fine-tuning conductor composer coil wire conductor conductor coil wire composer music Source-domain NMT model Target-domain NMT model Figure 1: Vocabulary adaptation for domain adaptation in NMT using cross-domain embedding projection. source-domain and small amount of target-domain parallel data, fine-tuning adjusts the parameters of a model pre-trained in the source-domain to the target domain. However, in fine-tuning, inheriting the embed- ding layers of the model pre-trained in the source domain causes vocabulary mismatches; namely, a model can handle neither domain-specific words that are not covered by a small amount of target- domain parallel data (unknown words) nor words that have different meanings across domains (se- mantic shift). Moreover, adopting the standard subword tokenization (Sennrich et al., 2016b; Kudo, 2018) accelerates the semantic shift. Target- domain-specific words are often finely decomposed into source-domain subwords (e.g., “alloy” → “ all” + …

Content truncated to 500 words