Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Findings of the Association for Computational Linguistics: ACL 2022, pages 700 - 710 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation Kushal Arora1∗ Layla El Asri2 Hareesh Bahuleyan3† Jackie Chi Kit Cheung1‡ 1Mila / McGill University 2Borealis AI 3 Zalando SE {kushal.arora@mail, jcheung@cs}.mcgill.ca layla.elasri@borealisai.com, hareeshbahuleyan@gmail.com Abstract Current language generation models suffer from issues such as repetition, incoherence, and hallucinations. An often-repeated hypoth- esis for this brittleness of generation models is that it is caused by the training and the gen- eration procedure mismatch, also referred to as exposure bias. In this paper, we verify this hypothesis by analyzing exposure bias from an imitation learning perspective. We show that exposure bias leads to an accumulation of er- rors during generation, analyze why perplexity fails to capture this accumulation of errors, and empirically show that this accumulation results in poor generation quality. 1 1 Introduction Large-scale neural language models have made great strides in a series of language generation tasks such as machine translation (Bahdanau et al., 2014; Vaswani et al., 2017; Raffel et al.), text sum- marization (See et al., 2017; Lewis et al., 2019; Zhang et al., 2019a), conversational dialog genera- tion (Serban et al., 2015; Lowe et al., 2017; Roller et al., 2020; Zhang et al., 2020), etc. However, despite the successes achieved by these models on several conditional generation tasks, they continue to suffer from degenerate be- haviors such as repetition, a lack of diversity, dull- ness, and, incoherence, especially in open-ended generation settings such as text completion and dialog modeling (Holtzman et al., 2019). This de- generate behavior is often attributed to a mismatch between the maximum likelihood training and gen- ∗A part of this work was done when the author was an intern at Borealis AI. †During a part of this work, the author was an employee at Borealis AI. ‡During a part of this work, the author was an Academic Advisor at Borealis AI. 1Source code to reproduce these experiments is available at https://github.com/kushalarora/ quantifying_exposure_bias. eration procedure mismatch (Welleck et al., 2019; Choi et al., 2020; Li et al., 2016). Maximum likelihood training also referred to as teacher forcing (Williams and Zipser, 1989), factorizes the language model as a linear chain, and maximizes the log-likelihood of this factorized language model on a training corpus. During this maximum likelihood training, the model learns a distribution of the next tokens conditioned on the contexts from the ground-truth training data. A concern with the MLE-based training is that the ground-truth contexts from the training corpus are not available during generation. Rather, the conditioning contexts during this phase comprise of tokens previously generated by the model itself. The distribution of these contexts seen during the generation phase might be very different from the ones encountered during the training phase. This mismatch is referred to as exposure bias (Ranzato et al., 2016; Bengio et al., 2015). A side-effect of exposure bias is that an error at …

Content truncated to 500 words