Autoregressive Text Generation Beyond Feedback Loops

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3400–3406, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3400 Autoregressive Text Generation Beyond Feedback Loops Florian Schmidt Department of Computer Science ETH Z¨urich florian.schmidt@inf.ethz.ch Stephan Mandt Department of Computer Science University of California, Irvine mandt@uci.edu Thomas Hofmann Department of Computer Science ETH Z¨urich thomas.hofmann@inf.ethz.ch Abstract Autoregressive state transitions, where predic- tions are conditioned on past predictions, are the predominant choice for both determinis- tic and stochastic sequential models. How- ever, autoregressive feedback exposes the evo- lution of the hidden state trajectory to po- tential biases from well-known train-test dis- crepancies. In this paper, we combine a la- tent state space model with a CRF observa- tion model. We argue that such autoregres- sive observation models form an interesting middle ground that expresses local correla- tions on the word level but keeps the state evo- lution non-autoregressive. On unconditional sentence generation we show performance im- provements compared to RNN and GAN base- lines while avoiding some prototypical failure modes of autoregressive models.1 1 Introduction Sequential autoregressive models express pre- dictions of observations based on past predic- tions. They are the predominant architecture for text generation in a maximum likelihood setup (Graves, 2013; Sutskever et al., 2014) and are used in machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), summarization (Rush et al., 2015), and dialogue systems (Serban et al., 2016). An immediate consequence of combining au- toregressive modeling and maximum likelihood training is that past observations enter the loss functions as ground-truth, not predicted observa- tions (Goodfellow et al., 2016). This discrepancy is often summarized as teacher-forcing and the bias it implies is referred to as exposure-bias (Ran- zato et al., 2016; Goyal et al., 2016). 1Code and generated sentences available at https:// github.com/schmiflo/crf-generation The standard methodology to turn a sequen- tial model into an autoregressive one is to intro- duce a feedback loop, where one provides the last predicted token as a feature to the computation of the next state (Graves, 2013). The ground- truth observations become effectively input fea- tures for the evolution of the hidden state trajec- tory at training time. Several attempts have been made to introduce robustness with respect to the model’s predictions by leaving the maximum like- lihood framework, either implicitly (Bengio et al., 2015; Bowman et al., 2016) or explicitly (Goyal et al., 2016; Leblond et al., 2018). Neverthe- less, the same feedback mechanisms have been adopted in latent sequential models where they obfuscate the true stochasticity of transitions dur- ing training. Non-autoregressive sequence mod- els have recently regained attention for uncondi- tional (Schmidt and Hofmann, 2018; M. Ziegler and M. Rush, 2019) and conditional (Lee et al., 2018) generation. We argue that there is an interesting interme- diate regime between feedback-driven autoregres- sive models and completely non-autoregressive models, namely modeling temporal correlations as part of the observation model. We propose a neu- ral CRF …

Content truncated to 500 words