Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870–6886 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6870 Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing Clara MeisterQ Elizabeth SaleskyZ Ryan CotterellD,Q QETH Z¨urich ZJohns Hopkins University DUniversity of Cambridge clara.meister@inf.ethz.ch esalesky@jhu.edu ryan.cotterell@inf.ethz.ch Abstract Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predic- tions, a common sign of overfitting. This class of techniques, of which label smoothing is one, has a connection to entropy regularization. De- spite the consistent success of label smoothing across architectures and datasets in language generation tasks, two problems remain open: (1) there is little understanding of the underly- ing effects entropy regularizers have on mod- els, and (2) the full space of entropy regulariza- tion techniques is largely unexplored. We in- troduce a parametric family of entropy regular- izers, which includes label smoothing as a spe- cial case, and use it to gain a better understand- ing of the relationship between the entropy of a trained model and its performance on language generation tasks. We also find that variance in model performance can be explained largely by the resulting entropy of the model. Lastly, we find that label smoothing provably does not allow for sparse distributions, an undesir- able property for language generation models, and therefore advise the use of other entropy regularization methods in its place. Our code is available online at https://github.com/ rycolab/entropyRegularization. 1 Introduction When training large neural networks with mil- lions of parameters, regularization of some form is needed to prevent overfitting, even when large amounts of data are used; models for language generation are no exception. In probabilistic mod- eling, e.g. when the final layer of the neural net- work is a softmax, overfitting often manifests it- self in overconfident placement of most of the probability mass on a few candidates, resulting in peaky (low-entropy) probability distributions over the vocabulary. Specifically for language gener- ation tasks, this behavior leads to the output of repetitive or frequently occurring but unrelated text, which is detrimental to the generalization abilities of the model (Chorowski and Jaitly, 2017; Holtz- man et al., 2020). A natural regularizer to consider is, therefore, one that penalizes overconfidence, encouraging higher entropy in the learned distri- bution. Indeed, the literature has ascribed gains of ≈ 1 BLEU point in machine translation to label smoothing, one such technique (Chen et al., 2018). Despite the clear relationship between low en- tropy and overfitting, only a handful of distinct entropy regularizers have been explored. To fill this gap, we introduce generalized entropy regu- larization (GER), a unified framework for under- standing and exploring a broad range of entropy- inducing regularizers. GER is based on the skew- Jensen family of divergences Jα,G (Nielsen and Boltz, 2011) and thus may be generalized to any Bregman divergence through the choice of genera- tor function G. For the negative entropy generator function, GER recovers label smoothing (Szegedy et …

Content truncated to 500 words