Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9781 - 9792 December 7-11, 2022 ©2022 Association for Computational Linguistics Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation Dongkyu Lee1,2∗ Ka Chun Cheung2 Nevin L. Zhang1 1Department of Computer Science and Engineering, HKUST 2NVIDIA AI Technology Center, NVIDIA dleear@cse.ust.hk chcheung@nvidia.com lzhang@cse.ust.hk Abstract Overconfidence has been shown to impair gen- eralization and calibration of a neural network. Previous studies remedy this issue by adding a regularization term to a loss function, pre- venting a model from making a peaked dis- tribution. Label smoothing smoothes target labels with a pre-defined prior label distribu- tion; as a result, a model is learned to maxi- mize the likelihood of predicting the soft la- bel. Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training. In other words, label smoothing does not reflect the change in probability dis- tribution mapped by a model over the course of training. To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by tak- ing model probability distribution into account, thereby varying the parameter per instance. A model in training self-regulates the extent of smoothing on the fly during forward propaga- tion. Furthermore, inspired by recent work in bridging label smoothing and knowledge dis- tillation, our work utilizes self-knowledge as a prior label distribution in softening target la- bels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter. Our regularizer is validated comprehensively, and the result illustrates marked improvements in model generalization and calibration, enhanc- ing robustness and trustworthiness of a model. 1 Introduction In common practice, a neural network is trained to maximize the expected likelihood of observed tar- gets, and the gradient with respect to the objective updates the learnable model parameters. With hard targets (one-hot encoded), the maximum objective can be approached when a model assigns a high probability mass to the corresponding target label ∗This work was done while Dongkyu was an intern at NVIDIA over the output space. That is, due to the normaliz- ing activation functions (i.e. softmax), a model is trained in order for logits to have a marked differ- ence between the target logit and the other classes logits (Müller et al., 2019). Despite its wide application and use, the max- imum likelihood estimation with hard targets has been found to incur an overconfident problem; the predictive score of a model does not reflect the ac- tual accuracy of the prediction. Consequently, this leads to degradation in model calibration (Pereyra et al., 2017), as well as in model performance (Müller et al., 2019). Additionally, this problem stands out more clearly with a limited number of samples, as a model is more prone to overfitting. To remedy such phenomenon, Szegedy et al. (2016) proposed label smoothing, in which one-hot en- coded targets are replaced with smoothed targets. Label smoothing has boosted performance in com- puter vision (Szegedy …

Content truncated to 500 words