Self-Paced Learning for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1074–1080, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1074 Self-Paced Learning for Neural Machine Translation Yu Wan†‡ Baosong Yang‡∗ Derek F. Wong†∗ Yikai Zhou† Lidia S. Chao† Haibo Zhang‡ Boxing Chen‡ †NLP2CT Lab, University of Macau nlp2ct.{ywan,yzhou}@gmail.com, {derekfw,lidiasc}@umac.mo ‡Alibaba Group {yangbaosong.ybs,zhanhui.zhb,boxing.cbx}@alibaba-inc.com Abstract Recent studies have proven that the training of neural machine translation (NMT) can be fa- cilitated by mimicking the learning process of humans. Nevertheless, achievements of such kind of curriculum learning rely on the quality of artificial schedule drawn up with the hand- crafted features, e.g. sentence length or word rarity. We ameliorate this procedure with a more flexible manner by proposing self-paced learning, where NMT model is allowed to 1) automatically quantify the learning confidence over training examples; and 2) flexibly govern its learning via regulating the loss in each it- eration step. Experimental results over multi- ple translation tasks demonstrate that the pro- posed model yields better performance than strong baselines and those models trained with human-designed curricula on both translation quality and convergence speed.1 1 Introduction Neural machine translation (NMT) has achieved promising results with the use of various optimiza- tion tricks (Hassan et al., 2018; Chen et al., 2018; Xu et al., 2019; Li et al., 2020; Yang et al., 2020). In spite of that, these techniques lead to increased training time and massive hyper-parameters, mak- ing the development of a well-performed system expensive (Popel and Bojar, 2018; Ott et al., 2018). As an alternative mitigation, curriculum learn- ing (CL, Elman, 1993; Bengio et al., 2009) has shown its effectiveness on speeding up the con- vergence and stabilizing the NMT model train- ing (Zhang et al., 2018; Platanios et al., 2019). CL teaches NMT model from easy examples to com- plex ones rather than equally considering all sam- ples, where the keys lie in the definition of “diffi- ∗Baosong Yang and Derek F. Wong are co-corresponding authors. Work was done when Yu Wan was interning at DAMO Academy, Alibaba Group. 1Our codes: https://github.com/NLP2CT/SPL for NMT. culty” and the strategy of curricula design (Krueger and Dayan, 2009; Kocmi and Bojar, 2017). Ex- isting studies artificially determine data difficulty according to prior linguistic knowledge such as sen- tence length (SL) and word rarity (WR) (Platanios et al., 2019; Zhang et al., 2019; Zhou et al., 2020), and manually tune the learning schedule (Liu et al., 2020; Fomicheva et al., 2020). However, neither there exists a clear distinction between easy and hard examples (Kumar et al., 2010), nor these hu- man intuitions exactly conform to effective model training (Zhang et al., 2018). Instead, we resolve this problem by introducing self-paced learning (Kumar et al., 2010), where the emphasis of learning can be dynamically deter- mined by model itself rather than human intuitions. Specifically, our model measures the level of confi- dence on each training example (Gal and Ghahra- mani, 2016; Xiao and Wang, 2019), where an easy sample is actually the one of …

Content truncated to 500 words