Competence-based Curriculum Learning for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1162–1172 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1162 Competence-based Curriculum Learning for Neural Machine Translation Emmanouil Antonios Platanios†, Otilia Stretcu†, Graham Neubig‡, Barnabas Poczos†, Tom M. Mitchell† †Machine Learning Department, ‡Language Technologies Institute Carnegie Mellon University {e.a.platanios,ostretcu,gneubig,bpoczos,tom.mitchell}@cs.cmu.edu Abstract Current state-of-the-art NMT systems use large neural networks that are not only slow to train, but also often require many heuristics and optimization tricks, such as specialized learning rate schedules and large batch sizes. This is undesirable as it requires extensive hy- perparameter tuning. In this paper, we propose a curriculum learning framework for NMT that reduces training time, reduces the need for spe- cialized heuristics or large batch sizes, and re- sults in overall better performance. Our frame- work consists of a principled way of deciding which training samples are shown to the model at different times during training, based on the estimated difficulty of a sample and the cur- rent competence of the model. Filtering train- ing samples in this manner prevents the model from getting stuck in bad local optima, mak- ing it converge faster and reach a better solu- tion than the common approach of uniformly sampling training examples. Furthermore, the proposed method can be easily applied to ex- isting NMT models by simply modifying their input data pipelines. We show that our frame- work can help improve the training time and the performance of both recurrent neural net- work models and Transformers, achieving up to a 70% decrease in training time, while at the same time obtaining accuracy improvements of up to 2.2 BLEU. 1 Introduction Neural Machine Translation (NMT; Kalchbrenner and Blunsom (2013); Bahdanau et al. (2015)) now represents the state-of-the-art adapted in most ma- chine translation systems (Wu et al., 2016; Crego et al., 2016; Bojar et al., 2017a), largely due to its ability to benefit from end-to-end training on massive amounts of data. In particular, recently- introduced self-attentional Transformer architec- tures (Vaswani et al., 2017) are rapidly becoming the de-facto standard in NMT, having demonstrated CURRICULUM LEARNING DIFFICULTY Use sample only if: difficulty(sample) ≤ competence(model) COMPETENCE MODEL TRAINER DATA SAMPLE MODEL STATE Figure 1: Overview of the proposed curriculum learn- ing framework. During training, difficulty of each train- ing sample is estimated and a decision whether to use it is made based on the current competence of the model. both superior performance and training speed com- pared to previous architectures using recurrent neu- ral networks (RNNs; (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014)). However, large scale NMT systems are often hard to train, requiring complicated heuristics which can be both time- consuming and expensive to tune. This is espe- cially true for Transformers which, when carefully tuned, have been shown to consistently outperform RNNs (Popel and Bojar, 2018), but on the other hand, also rely on a number of heuristics such as specialized learning rates and large-batch training. In this paper, we attempt to tackle this problem by proposing a curriculum …

Content truncated to 500 words