Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1115–1124 November 20–23, 2022. ©2022 Association for Computational Linguistics 1115 Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling Ilia Kulikov∗ New York University kulikov@cs.nyu.edu Maksim Eremeev∗ New York University eremeev@nyu.edu Kyunghyun Cho New York University Genentech CIFAR Fellow in LMB Abstract Neural autoregressive sequence models smear the probability among many possible sequences including degenerate ones, such as empty or repetitive sequences. In this work, we tackle one specific case where the model assigns a high probability to unreasonably short se- quences. We define the oversmoothing rate to quantify this issue. After confirming the high degree of oversmoothing in neural machine translation, we propose to explicitly minimize the oversmoothing rate during training. We conduct a set of experiments to study the effect of the proposed regularization on both model distribution and decoding performance. We use a neural machine translation task as the testbed and consider three different datasets of vary- ing size. Our experiments reveal three major findings. First, we can control the oversmooth- ing rate of the model by tuning the strength of the regularization. Second, by enhancing the oversmoothing loss contribution, the prob- ability and the rank of ⟨eos⟩ token decrease heavily at positions where it is not supposed to be. Third, the proposed regularization im- pacts the outcome of beam search especially when a large beam is used. The degradation of translation quality (measured in BLEU) with a large beam significantly lessens with lower oversmoothing rate, but the degradation com- pared to smaller beam sizes remains to exist. From these observations, we conclude that the high degree of oversmoothing is the main rea- son behind the degenerate case of overly prob- able short sequences in a neural autoregressive model. 1 Introduction Neural autoregressive sequence modeling is a widely used scheme for conditional text genera- tion. It is applied to many NLP tasks, including machine translation, language modeling, and con- versation modeling (Cho et al., 2014; Sutskever ∗Equal contribution. et al., 2014; Brown et al., 2020; Roller et al., 2021). Despite the substantial success, major issues still exist, and it is still an active area of research. Here we highlight two major issues which have been discussed extensively. The first issue is the model assigning too high a probability to a sequence which is unreasonably shorter than a ground-truth sequence. Stahlberg and Byrne (2019) report evidence of an extreme case where the model frequently assigns the highest probability to an empty sequence given a source se- quence in machine translation. In addition, Koehn and Knowles (2017) demonstrate that the length of generated translation gets shorter with better decoding (i.e., beam search with a larger beam.) In the second issue, which is more often observed in open-ended sequence generation tasks, such as sequence completion, generated se- quences often contain unreasonably many repeti- tions (Holtzman et al., …

Content truncated to 500 words