Online Distilling from Checkpoints for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1932–1941 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1932 Online Distilling from Checkpoints for Neural Machine Translation Hao-Ran Wei, Shujian Huang ∗, Ran Wang, Xin-yu Dai and Jiajun Chen State Key Laboratory for Novel Software Technology Nanjing University Nanjing 210023, China {weihr, huangsj, wangr, daixy, chenjj}@nlp.nju.edu.cn Abstract Current predominant neural machine transla- tion (NMT) models often have a deep struc- ture with large amounts of parameters, mak- ing these models hard to train and easily suf- fering from over-fitting. A common prac- tice is to utilize a validation set to evaluate the training process and select the best check- point. Average and ensemble techniques on checkpoints can lead to further performance improvement. However, as these methods do not affect the training process, the system per- formance is restricted to the checkpoints gen- erated in the original training procedure. In contrast, we propose an online knowledge dis- tillation method. Our method on-the-fly gener- ates a teacher model from checkpoints, guid- ing the training process to obtain better per- formance. Experiments on several datasets and language pairs show steady improvement over a strong self-attention-based baseline sys- tem. We also provide analysis on data-limited setting against over-fitting. Furthermore, our method leads to an improvement on a machine reading experiment as well. 1 Introduction Neural Machine Translation (NMT) (Cho et al., 2014; Sutskever et al., 2014) has been rapidly de- veloped during the past several years. For further performance improvement, deeper and more ex- pressive structures (Johnson et al., 2017; Barone et al., 2017b; Gehring et al., 2017; Vaswani et al., 2017) have been exploited. However, all of these models have more than hundreds of millions of pa- rameters, which makes the training process more challenging. During the training of NMT models, we notice the following two problematic phenom- ena: First, the training process is unstable. This is evidenced by the decreasing of training loss with ∗Corresponding Author. fluctuate performance on the validation set. Second, the performance on validation set usually begins to worsen after several epochs, while the training loss keeps decreasing, which suggests the model being at risk of over-fitting. In order to alleviate these issues, the common practice is to periodically evaluate models on a held-out set (with each evaluated model saved as a checkpoint). Training is terminated when m con- secutive checkpoints show no improvement and select the checkpoint with best evaluation score as the final model. Further improvement can be achieved by utilizing more checkpoints, by smooth- ing, which averages these checkpoints’ parameters to generate more desirable parameters (Sennrich et al., 2016a); or by ensemble, which averages these checkpoints’ output probabilities at every step dur- ing inference (Chen et al., 2017). However, we notice that all of these methods have a limitation. Once the training process gets pa- rameters with poor performance, selecting, smooth- ing or ensemble from the checkpoints in this pro- cess may have limited generalization performance as well. We impute the limitation to the “offline” …

Content truncated to 500 words