Towards a Better Understanding of Label Smoothing in Neural Machine Translation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 212–223 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 212 Towards a Better Understanding of Label Smoothing in Neural Machine Translation Yingbo Gao Weiyue Wang Christian Herold Zijian Yang Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {gao|wwang|herold|zyang|ney}@i6.informatik.rwth-aachen.de Abstract In order to combat overfitting and in pur- suit of better generalization, label smoothing is widely applied in modern neural machine translation systems. The core idea is to penal- ize over-confident outputs and regularize the model so that its outputs do not diverge too much from some prior distribution. While training perplexity generally gets worse, label smoothing is found to consistently improve test performance. In this work, we aim to better understand label smoothing in the con- text of neural machine translation. Theoreti- cally, we derive and explain exactly what la- bel smoothing is optimizing for. Practically, we conduct extensive experiments by varying which tokens to smooth, tuning the probability mass to be deducted from the true targets and considering different prior distributions. We show that label smoothing is theoretically well- motivated, and by carefully choosing hyperpa- rameters, the practical performance of strong neural machine translation systems can be fur- ther improved. 1 Introduction In recent years, Neural Network (NN) models bring steady and concrete improvements on the task of Machine Translation (MT). From the introduction of sequence-to-sequence models (Cho et al., 2014; Sutskever et al., 2014a), to the invention of the at- tention mechanism (Bahdanau et al., 2015; Luong et al., 2015), end-to-end sequence learning with attention becomes the dominant design choice for Neural Machine Translation (NMT) models. From the study of convolutional sequence to sequence learning (Gehring et al., 2017a,b), to the prosperity of self-attention networks (Vaswani et al., 2017; Devlin et al., 2019), modern NMT systems, es- pecially Transformer-based ones (Vaswani et al., 2017), often deliver state-of-the-art performances (Bojar et al., 2018; Barrault et al., 2019), even un- der the condition of large-scale corpora (Ott et al., 2018; Edunov et al., 2018). In Transformer-based models, label smoothing is a widely applied method to improve model per- formance. Szegedy et al. (2016) initially introduce the method when making refinements to the Incep- tion (Szegedy et al., 2015) model, with the mo- tivation to combat overfitting and improve adapt- ability. In principle, label smoothing discounts a certain probability mass from the true label and redistributes it uniformly across all the class la- bels. This lowers the difference between the largest probability output and the others, effectively dis- couraging the model to generate overly confident predictions. Since information entropy (Shannon, 1948) can be thought of as a confidence measure of a probability distribution, Pereyra et al. (2017) add a negative entropy regularization term to the conventional cross entropy training criterion and compare it with uniform smoothing and unigram smoothing. M¨uller et al. (2019) …

Content truncated to 500 words