MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9027–9050 July 9-14, 2023 ©2023 Association for Computational Linguistics MIXCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies Shiyue Zhang♠∗ Shijie Wu♡ Ozan ˙Irsoy♡ Steven Lu♡ Mohit Bansal♠ Mark Dredze♡♣ David Rosenberg♡ ♡Bloomberg ♠UNC Chapel Hill ♣Johns Hopkins University Abstract Autoregressive language models are trained by minimizing the cross-entropy of the model dis- tribution Qθ relative to the data distribution P – that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood es- timation (MLE). We have observed that models trained in this way may “over-generalize”, in the sense that they produce non-human-like text. Moreover, we believe that reverse cross- entropy, i.e., the cross-entropy of P relative to Qθ, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MIXCE, an objec- tive that mixes the forward and reverse cross- entropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the result- ing models yield better generated text without complex decoding strategies. https://github.com/bloomberg/ mixce-acl2023 1 Introduction Rapid advances in pre-trained large-scale autore- gressive language models (LMs) have dramati- cally improved the performance of a variety of tasks (Radford et al., 2019; Brown et al., 2020; Zhang et al., 2022; Chowdhery et al., 2022). How- ever, these systems still struggle in many open- ended generation settings, where they are asked to produce a long text following a short prompt. In these cases, we seek systems that generate sen- sical, coherent, fluent, and engaging, or in short, human-like text (Pillutla et al., 2022). Different decoding strategies to generate such text from pretrained LMs suffer from different de- generation problems. Unbiased sampling1 usually ∗ Work done during an internship at Bloomberg. 1Unbiased sampling is vanilla random sampling, i.e., sam- pling with temperature=1.0. It is also called ancestral sam- pling (Eikema and Aziz, 2020) or pure sampling (Holtzman P P Q Q -logP(x) -logQ(x) Reverse Cross-Entropy Forward Cross-Entropy x ~ Q x ~ P Figure 1: MIXCE combines two complementary driving forces: reverse CE helps narrow the model distribution Qθ down when it is broader than data distribution P, while forward CE helps broaden Qθ out when it is nar- rower than P.2 results in incoherent and nonsensical text, while greedy and beam searches often get stuck in repeti- tion loops (Holtzman et al., 2020). These observa- tions suggest that the learned LM distribution Qθ still differs substantially from the human LM distri- bution P. A possible reason is that the autoregres- sive modeling of Qθ gives a non-zero probability to every possible sequence of tokens, while many se- quences are impossible under P. Nevertheless, we still hope that Qθ(x) is as small as possible when P(x) = 0. To this end, maximum likelihood esti- mation (MLE), i.e., minimizing the cross-entropy (CE) −Ex∼P [log Qθ(x)], is the most widely used objective …

Content truncated to 500 words