Uncertainty-Aware Curriculum Learning for Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6934–6944 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6934 Uncertainty-Aware Curriculum Learning for Neural Machine Translation Yikai Zhou† Baosong Yang‡ Derek F. Wong†∗ Yu Wan† Lidia S. Chao† †NLP2CT Lab, Department of Computer and Information Science, University of Macau nlp2ct.{yzhou,ywan}@gmail.com, {derekfw,lidiasc}@um.edu.mo ‡Alibaba DAMO Academy yangbaosong.ybs@alibaba-inc.com Abstract Neural machine translation (NMT) has proven to be facilitated by curriculum learning which presents examples in an easy-to-hard order at different training stages. The keys lie in the assessment of data difficulty and model com- petence. We propose uncertainty-aware cur- riculum learning, which is motivated by the intuition that: 1) the higher the uncertainty in a translation pair, the more complex and rarer the information it contains; and 2) the end of the decline in model uncertainty indicates the completeness of current training stage. Specif- ically, we serve cross-entropy of an example as its data difficulty and exploit the variance of distributions over the weights of the network to present the model uncertainty. Extensive ex- periments on various translation tasks reveal that our approach outperforms the strong base- line and related methods on both translation quality and convergence speed. Quantitative analyses reveal that the proposed strategy of- fers NMT the ability to automatically govern its learning schedule. 1 Introduction Neural machine translation (NMT) has advanced the state-of-the-art on various translation tasks (Hassan et al., 2018; Chen et al., 2018). A well- performed NMT is trained using an end-to-end framework (Sutskever et al., 2014) that profits from large-scale training corpus and various optimiza- tion tricks (Ott et al., 2018; Xu et al., 2019; Li et al., 2020). These techniques boost the transla- tion quality, in the meanwhile, leading to massive hyper-parameters to be tuned and expensive devel- opment costs (Popel and Bojar, 2018). Recent stud- ies (Zhang et al., 2018, 2019; Platanios et al., 2019; Liu et al., 2020) have proven that feeding training examples in a meaningful order rather than con- sidering them randomly can accelerate the model ∗Corresponding author Figure 1: The change of confidence in an area dur- ing the learning. Humans (red) experience the process of overconfidence⇒despair⇒enlightenment (Dunning- Kruger Curve), while prior work that exploits CL in NMT assumes a monotonically increased curve (green, Platanios et al., 2019). Interestingly, our model auto- matically draws a similar tendency as humans (blue). convergence thus reducing the computational cost. Such methods refer to curriculum learning (CL, Bengio et al., 2009), in which a model is taught as a human from simple concepts to complex ones. There exists two open problems in the integra- tion of CL with NMT, i.e. the assessment of data difficulty and the programme of learning schedule. Considering the former, prior studies (Kocmi and Bojar, 2017; Platanios et al., 2019) intuitively treat human linguistic knowledge, e.g. either sentence length or word rarity, as the measure of difficulty. Nevertheless, each linguistic feature merely consid- ers an aspect of sentences which fails to fully cope with the …

Content truncated to 500 words