Quality Signals in Generated Stories

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 192–202 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Quality Signals in Generated Stories Manasvi Sagarkar∗ John Wieting† Lifu Tu‡ Kevin Gimpel‡ ∗University of Chicago, Chicago, IL, 60637, USA †Carnegie Mellon University, Pittsburgh, PA, 15213, USA ‡Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA manasvi@uchicago.edu, jwieting@cs.cmu.edu, {lifu,kgimpel}@ttic.edu Abstract We study the problem of measuring the qual- ity of automatically-generated stories. We fo- cus on the setting in which a few sentences of a story are provided and the task is to generate the next sentence (“continuation”) in the story. We seek to identify what makes a story con- tinuation interesting, relevant, and have high overall quality. We crowdsource annotations along these three criteria for the outputs of story continuation systems, design features, and train models to predict the annotations. Our trained scorer can be used as a rich feature function for story generation, a reward func- tion for systems that use reinforcement learn- ing to learn to generate stories, and as a partial evaluation metric for story generation. 1 Introduction We study the problem of automatic story gen- eration in the climate of neural network natu- ral language generation methods. Story genera- tion (Mani, 2012; Gerv´as, 2012) has a long his- tory, beginning with rule-based systems in the 1970s (Klein et al., 1973; Meehan, 1977). Most story generation research has focused on mod- eling the plot, characters, and primary action of the story, using simplistic methods for producing the actual linguistic form of the stories (Turner, 1993; Riedl and Young, 2010). More recent work learns from data how to generate stories holisti- cally without a clear separation between content selection and surface realization (McIntyre and Lapata, 2009), with a few recent methods based on recurrent neural networks (Roemmele and Gor- don, 2015; Huang et al., 2016). We follow the latter style and focus on a setting in which a few sentences of a story are provided (the context) and the task is to generate the next sentence in the story (the continuation). Our goal is to produce continuations that are both interest- ing and relevant given the context. Neural networks are increasingly employed for natural language generation, most often with encoder-decoder architectures based on recurrent neural networks (Cho et al., 2014; Sutskever et al., 2014). However, while neural methods are effec- tive for generation of individual sentences condi- tioned on some context, they struggle with coher- ence when used to generate longer texts (Kiddon et al., 2016). In addition, it is challenging to apply neural models in less constrained generation tasks with many valid solutions, such as open-domain dialogue and story continuation. The story continuation task is difficult to formu- late and evaluate because there can be a wide va- riety of reasonable continuations for typical story contexts. This is also the case in open-domain dia- logue systems, in which common evaluation met- rics like BLEU (Papineni et al., 2002) are only weakly correlated …

Content truncated to 500 words