DEEP: DEnoising Entity Pre-training for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1753 - 1766 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DEEP: DEnoising Entity Pre-training for Neural Machine Translation Junjie Hu1, Hiroaki Hayashi3, Kyunghyun Cho2, Graham Neubig3 1University of Wisconsin-Madison, 2New York University, 3Carnegie Mellon University junjie.hu@wisc.edu, kyunghyun.cho@nyu.edu {hiroakih,gneubig}@cs.cmu.edu Abstract It has been shown that machine translation models usually generate poor translations for named entities that are infrequent in the train- ing corpus. Earlier named entity translation methods mainly focus on phonetic transliter- ation, which ignores the sentence context for translation and is limited in domain and lan- guage coverage. To address this limitation, we propose DEEP, a DEnoising Entity Pre- training method that leverages large amounts of monolingual data and a knowledge base to improve named entity translation accuracy within sentences. Besides, we investigate a multi-task learning strategy that finetunes a pre-trained neural machine translation model on both entity-augmented monolingual data and parallel data to further improve entity trans- lation. Experimental results on three language pairs demonstrate that DEEP results in signifi- cant improvements over strong denoising auto- encoding baselines, with a gain of up to 1.3 BLEU and up to 9.2 entity accuracy points for English-Russian translation.1 1 Introduction Proper translation of named entities is critically im- portant for accurately conveying the content of text in a number of domains, such as news or encyclope- dic text (Knight and Graehl, 1998; Al-Onaizan and Knight, 2002a,b). In addition, a growing number of new named entities (e.g., person name, location) appear every day, therefore many of these entities may not exist in the parallel data traditionally used to train MT systems. As a result, even state-of-the- art MT systems struggle with entity translation. For example, Laubli et al. (2020) note that a Chinese- English news translation system that had allegedly reached human parity still lagged far behind hu- man translators on entity translations, and this prob- 1Code/data/models are released at https://github. com/JunjieHu/deep. lem will be further exacerbated in the cross-domain transfer settings or in the case of emerging entities. Because of this, there have been a number of methods proposed specifically to address the prob- lem of translating entities. As noted by Liu (2015), earlier studies on named entity translation largely focused on rule-based methods (Wan and Verspoor, 1998), statistical alignment methods (Huang et al., 2003, 2004) and Web mining methods (Huang et al., 2005; Wu and Chang, 2007; Yang et al., 2009). However, these methods have two main issues. First, as they generally translate a single named entity without any context in a sentence, it makes it diffi- cult to resolve ambiguity in entities using context. In addition, the translation of entities is often per- formed in a two-step process of entity recognition then translation, which complicates the translation pipeline and can result in cascading errors (Huang et al., 2003, 2004; Chen et al., 2013). In this paper, we focus on a simple yet effec- tive method that improves named entity translation …

Content truncated to 500 words