Globally Coherent Text Generation with Neural Checklist Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 329–339, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Globally Coherent Text Generation with Neural Checklist Models Chlo´e Kiddon Luke Zettlemoyer Yejin Choi Computer Science & Engineering University of Washington {chloe, lsz, yejin}@cs.washington.edu Abstract Recurrent neural networks can generate lo- cally coherent text but often have difficulties representing what has already been generated and what still needs to be said – especially when constructing long texts. We present the neural checklist model, a recurrent neural net- work that models global coherence by stor- ing and updating an agenda of text strings which should be mentioned somewhere in the output. The model generates output by dy- namically adjusting the interpolation among a language model and a pair of attention mod- els that encourage references to agenda items. Evaluations on cooking recipes and dialogue system responses demonstrate high coherence with greatly improved semantic coverage of the agenda. 1 Introduction Recurrent neural network (RNN) architectures have proven to be well suited for many natural language generation tasks (Mikolov et al., 2010; Mikolov et al., 2011; Sordoni et al., 2015; Xu et al., 2015; Wen et al., 2015; Mei et al., 2016). Previous neu- ral generation models typically generate locally co- herent language that is on topic; however, overall they can miss information that should have been in- troduced or introduce duplicated or superfluous con- tent. These errors are particularly common in situ- ations where there are multiple distinct sources of input or the length of the output text is sufficiently long. In this paper, we present a new recurrent neu- ral model that maintains coherence while improv- Place tomatoes in a bowl . Dice the onion Pico de gallo chopped tomatoes onion jalapeños salt lime and add to the tomatoes ✓ ✓✓ … Figure 1: Example checklist recipe generation. A checklist (right dashed column) tracks which agenda items (top boxes; “salt,” “lime,” etc.) have already been used (checked boxes). The model is trained to interpolate an RNN (e.g., encode “pico de gallo” and decode a recipe) with attention models over new (left column) and used (middle column) items that identify likely items for each time step (shaded boxes; “tomatoes,” etc.). ing coverage by globally tracking what has been said and what is still left to be said in complete texts. For example, consider the challenge of generat- ing a cooking recipe, where the title and ingredient list are provided as inputs and the system must gen- erate a complete text that describes how to produce the desired dish. Existing RNN models may lose track of which ingredients have already been men- tioned, especially during the generation of a long recipe with many ingredients. Recent work has fo- cused on adapting neural network architectures to improve coverage (Wen et al., 2015) with applica- tion to generating customer service responses, such as hotel information, where a single sentence is gen- erated to describe a few key ideas. Our focus is …

Content truncated to 500 words