Self-Guided Curriculum Learning for Neural Machine Translation

Proceedings of the 18th International Conference on Spoken Language Translation, pages 206–214 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 206 Self-Guided Curriculum Learning for Neural Machine Translation Lei Zhou∗ Nagoya University zhou.lei@a.mbox.nagoya-u.ac.jp Liang Ding The University of Sydney ldin3097@sydney.edu.au Kevin Duh Johns Hopkins University kevinduh@cs.jhu.edu Shinji Watanabe Carnegie Mellon University shinjiw@ieee.org Ryohei Sasano Nagoya University sasano@i.nagoya-u.ac.jp Koichi Takeda Nagoya University takedasu@i.nagoya-u.ac.jp Abstract In supervised learning, a well-trained model should be able to recover ground truth ac- curately, i.e. the predicted labels are ex- pected to resemble the ground truth labels as much as possible. Inspired by this, we for- mulate a difficulty criterion based on the re- covery degrees of training examples. Mo- tivated by the intuition that after skimming through the training corpus, the neural ma- chine translation (NMT) model “knows” how to schedule a suitable curriculum according to learning difficulty, we propose a self-guided curriculum learning strategy that encourages the NMT model to learn from easy to hard on the basis of recovery degrees. Specifi- cally, we adopt sentence-level BLEU score as the proxy of recovery degree. Experimen- tal results on translation benchmarks includ- ing WMT14 English⇒German and WMT17 Chinese⇒English demonstrate that our pro- posed method considerably improves the re- covery degree, thus consistently improving the translation performance. 1 Introduction Inspired by the learning behavior of humans, Cur- riculum Learning (CL) for neural network training starts from a basic idea of “starting small”, namely better to start from easier aspects of a task and then progress towards aspects with increasing level of difficulty (Elman, 1993). Bengio et al. (2009) achieves significant performance boost on several tasks by forcing models to learn training examples following an order from “easy” to “difficult”. They further explain CL method with two important con- stituents: how to rank training examples by learn- ing difficulty and how to schedule the presentation of training examples based on that rank. ∗Part of this work was done when the first author visited CLSP, JHU. ̂y1 ̂y2 y1 y2 x1 x2 ̂y1 ̂y2 NMT NMT Figure 1: The NMT model is well-trained on parallel corpus D, {(x1, y1), (x2, y2)} ∈ D. ˆyi is translated from xi. The distance between the ground truth yi and the NMT generated hypothesis ˆyi represents the recovery degree (dashed arrows), which is computed by sentence-level BLEU in our case. Blue- and green- colored examples represent the NMT learned distribu- tion and the empirical distribution, respectively. Taking x1 and x2 as the input, the training example (x1, y1) shows a better recovery degree, which means it’s easier to be mastered than (x2, y2). In the field of neural machine translation (NMT), empirical studies have shown that CL strategies contribute to both convergence speed and model performance (Zhang et al., 2018; Platanios et al., 2019; Zhang et al., 2019; Liu et al., 2020; Zhan et al., 2021; Ruiter et al., 2020). These CL strate- gies vary by difficulty criteria and curriculum schedules. Early difficulty criterion depends on manually crafted features …

Content truncated to 500 words