Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2840–2850 August 1–6, 2021. ©2021 Association for Computational Linguistics 2840 Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation Wenxiang Jiao†∗ Xing Wang‡ Zhaopeng Tu‡ Shuming Shi‡ Michael R. Lyu† Irwin King† †Department of Computer Science and Engineering The Chinese University of Hong Kong, HKSAR, China ‡Tencent AI Lab †{wxjiao,lyu,king}@cse.cuhk.edu.hk ‡{brightxwang,zptu,shumingshi}@tencent.com Abstract Self-training has proven effective for improv- ing NMT performance by augmenting model training with synthetic parallel data. The common practice is to construct synthetic data based on a randomly sampled subset of large-scale monolingual data, which we em- pirically show is sub-optimal. In this work, we propose to improve the sampling proce- dure by selecting the most informative mono- lingual sentences to complement the paral- lel data. To this end, we compute the un- certainty of monolingual sentences using the bilingual dictionary extracted from the paral- lel data. Intuitively, monolingual sentences with lower uncertainty generally correspond to easy-to-translate patterns which may not pro- vide additional gains. Accordingly, we de- sign an uncertainty-based sampling strategy to efficiently exploit the monolingual data for self-training, in which monolingual sentences with higher uncertainty would be sampled with higher probability. Experimental results on large-scale WMT English⇒German and English⇒Chinese datasets demonstrate the ef- fectiveness of the proposed approach. Ex- tensive analyses suggest that emphasizing the learning on uncertain monolingual sentences by our approach does improve the translation quality of high-uncertainty sentences and also benefits the prediction of low-frequency words at the target side.1 1 Introduction Leveraging large-scale unlabeled data has become an effective approach for improving the perfor- mance of natural language processing (NLP) mod- els (Devlin et al., 2019; Brown et al., 2020; Jiao et al., 2020a). As for neural machine translation ∗Work was mainly done when Wenxiang Jiao was intern- ing at Tencent AI Lab. 1The source code is available at https://github. com/wxjiao/UncSamp (NMT), compared to the parallel data, the mono- lingual data is available in large quantities for many languages. Several approaches on boosting the NMT performance with the monolingual data have been proposed, e.g., data augmentation (Sen- nrich et al., 2016a; Zhang and Zong, 2016), semi- supervised training (Cheng et al., 2016; Zhang et al., 2018; Cai et al., 2021), pre-training (Siddhant et al., 2020; Liu et al., 2020). Among them, data augmentation with the synthetic parallel data (Sen- nrich et al., 2016a; Edunov et al., 2018) is the most widely used approach due to its simple and effec- tive implementation. It has been a de-facto standard in developing the large-scale NMT systems (Has- san et al., 2018; Ng et al., 2019; Wu et al., 2020; Huang et al., 2021). Self-training (Zhang and Zong, 2016) is one of the most commonly used approaches for data aug- mentation. Generally, self-training is performed in three steps: (1) randomly sample a subset from the large-scale monolingual data; (2) use a “teacher” NMT model to translate the subset data into …

Content truncated to 500 words