Dynamic Curriculum Learning for Low-Resource Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 3977–3989 Barcelona, Spain (Online), December 8-13, 2020 3977 Dynamic Curriculum Learning for Low-Resource Neural Machine Translation Chen Xu1, Bojie Hu2, Yufan Jiang3, Kai Feng1, Zeyang Wang1, Shen Huang2, Qi Ju2∗, Tong Xiao1,4, Jingbo Zhu1,4∗ 1NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China 2Tencent Minority-Mandarin Translation 3Smart Platform Product Department of Tencent Inc., China 4NiuTrans Research, Shenyang, China xuchenneu@163.com, {xiaotong,zhujingbo}@mail.neu.edu.cn {bojiehu, garyyfjiang, springhuang, damonju}@tencent.com Abstract Large amounts of data has made neural machine translation (NMT) a big success in recent years. But it is still a challenge if we train these models on small-scale corpora. In this case, the way of using data appears to be more important. Here, we investigate the effective use of training data for low-resource NMT. In particular, we propose a dynamic curriculum learning (DCL) method to reorder training samples in training. Unlike previous work, we do not use a static scoring function for reordering. Instead, the order of training samples is dynamically determined in two ways - loss decline and model competence. This eases training by highlighting easy samples that the current model has enough competence to learn. We test our DCL method in a Transformer- based system. Experimental results show that DCL outperforms several strong baselines on three low-resource machine translation benchmarks and different sized data of WMT’16 En-De. 1 Introduction Recently, neural machine translation (NMT) has demonstrated impressive performance improvements and became the de-facto standard (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016; Vaswani et al., 2017). However, like other neural methods, NMT is data-hungry. This makes it challenging when we train such a model in low-resource scenarios (Koehn and Knowles, 2017). Researchers have developed promising approaches to low-resource NMT. Among these are data augmentation (Sennrich et al., 2016a; Cheng et al., 2016; Fadaee et al., 2017), transfer learning (Zoph et al., 2016; Kocmi and Bojar, 2018), and pre-trained models (Peters et al., 2018; Devlin et al., 2019). But these approaches rely on external data other than bi-text. To date, it is rare to see work on the effective use of bilingual data for low-resource NMT. In general, the way of feeding samples plays an important role in training neural models. A good instance is that it is popular to shuffle the input data for robust training in state-of-the-art systems. More systematic studies on this issue can be found in recent papers (Bengio et al., 2009; Kumar et al., 2010; Shrivastava et al., 2016). For example, Arpit et al. (2017) have pointed out that deep neural networks tend to prioritize learning “easy” samples first. This agrees with the idea of curriculum learning (Bengio et al., 2009) in that an easy-to-hard learning strategy can yield better convergence for training. In NMT, curriculum learning is not new. Several research groups have applied it to large-scale transla- tion tasks although few of them discuss the issue in a low-resource setup (Zhang et al., 2018; Platanios et al., 2019; Liu …

Content truncated to 500 words