Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2255–2266, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2255 Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation Wenxiang Jiao†∗ Xing Wang‡ Shilin He† Irwin King† Michael R. Lyu† Zhaopeng Tu‡ †Department of Computer Science and Engineering The Chinese University of Hong Kong, HKSAR, China ‡Tencent AI Lab †{wxjiao,slhe,king,lyu}@cse.cuhk.edu.hk ‡{brightxwang,zptu}@tencent.com Abstract Large-scale training datasets lie at the core of the recent success of neural machine transla- tion (NMT) models. However, the complex patterns and potential noises in the large-scale data make training NMT models difficult. In this work, we explore to identify the inactive training examples which contribute less to the model performance, and show that the exis- tence of inactive examples depends on the data distribution. We further introduce data rejuve- nation to improve the training of NMT mod- els on large-scale datasets by exploiting inac- tive examples. The proposed framework con- sists of three phases. First, we train an iden- tification model on the original training data, and use it to distinguish inactive examples and active examples by their sentence-level output probabilities. Then, we train a rejuvenation model on the active examples, which is used to re-label the inactive examples with forward- translation. Finally, the rejuvenated examples and the active examples are combined to train the final NMT model. Experimental results on WMT14 English-German and English-French datasets show that the proposed data rejuve- nation consistently and significantly improves performance for several strong NMT models. Extensive analyses reveal that our approach stabilizes and accelerates the training process of NMT models, resulting in final models with better generalization capability. 1 1 Introduction Neural machine translation (NMT) is a data-hungry approach, which requires a large amount of data to train a well-performing NMT model (Koehn and Knowles, 2017). However, the complex pat- terns and potential noises in the large-scale data ∗Work was mainly done when Wenxiang Jiao and Shilin He were interning at Tencent AI Lab. 1The source code is available at https://github. com/wxjiao/Data-Rejuvenation make training NMT models difficult. To relieve this problem, several approaches have been pro- posed to better exploit the training data, such as curriculum learning (Platanios et al., 2019), data diversification (Nguyen et al., 2019), and data de- noising (Wang et al., 2018). In this paper, we explore an interesting alterna- tive which is to reactivate the inactive examples in the training data for NMT models. By definition, in- active examples are the training examples that only marginally contribute to or even inversely harm the performance of NMT models. Concretely, we use sentence-level output probability (Kumar and Sarawagi, 2019) assigned by a trained NMT model to measure the activeness level of training exam- ples, and regard the examples with the least proba- bilities as inactive examples (§3.1). Experimental results show that removing 10% most inactive ex- amples can marginally improve translation perfor- mance. In addition, we observe a high overlapping ratio (e.g., around 80%) of the most inactive and active examples across …

Content truncated to 500 words