Toward Better Storylines with Sentence-Level Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7472–7478 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7472 Toward Better Storylines with Sentence-Level Language Models Daphne Ippolito†∗ daphnei@seas.upenn.edu David Grangier† grangier@google.com Douglas Eck† deck@google.com Chris Callison-Burch†* ccb@seas.upenn.edu Abstract We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi- sentence coherence. Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embed- dings and predicts an embedding for the next sentence, which is more efficient than predict- ing word embeddings. Notably this allows us to consider a large number of candidates for the next sentence during training. We demon- strate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks. 1 Introduction Computer generation of stories and other kinds of creative writing is a challenging endeavor. It en- tangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline. In the recent year, neural language mod- els have made tremendous progress with respect to fluency (Bahdanau et al., 2015; Vaswani et al., 2017; Bengio et al., 2003; Devlin et al., 2019), but coherency is still a major challenge (See et al., 2019). The generation of coherent stories has re- cently been addressed with additional conditioning: Fan et al. (2018) suggest conditioning on a story prompt, Clark et al. (2018) propose collaboration between a generative model and a human writer, and Guan et al. (2019) suggest attending to a com- monsense graph relevant to the story plot. Con- ditioning based on a generated story plan (Martin et al., 2018; Fan et al., 2019; Yao et al., 2019), a se- ∗University of Pennsylvania, †Google quence of images (Chandu et al., 2019) or character roles (Liu et al., 2020) have also been considered. Our work is orthogonal to these efforts. Rather than considering additional conditioning, we pro- pose a model which takes as input several sentences of context and selects the best next sentence within a large set of fluent candidate sentences. We lever- age pre-trained BERT embeddings (Devlin et al., 2019) to build this sentence-level language model. Given the embeddings of the previous sentences of the story, our model learns to predict a likely embedding of the next sentence. This task isolates the modeling of long-range dependencies from the prediction of individual words, which has several advantages. First, since our model only needs to determine how well each candidate sentence would fit as a coherent con- tinuation to the story, it does not spend capacity and time to learn fluency. Second, our model does not manipulate individual words but full sentences, which allows us to consider tens of thousands of candidate sentences at …

Content truncated to 500 words