Do Massively Pretrained Language Models Make Better Storytellers?

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 843–861 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 843 Do Massively Pretrained Language Models Make Better Storytellers? Abigail See, Aneesh Pappu∗, Rohun Saxena∗, Akhila Yerukola∗, Christopher D. Manning Stanford University {abisee,apappu,rohun,akhilay,manning}@cs.stanford.edu Abstract Large neural language models trained on mas- sive amounts of text have emerged as a formidable strategy for Natural Language Un- derstanding tasks. However, the strength of these models as Natural Language Generators is less clear. Though anecdotal evidence sug- gests that these models generate better quality text, there has been no detailed study charac- terizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Rad- ford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018). By evaluating the generated text across a wide va- riety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers. We find that al- though GPT2-117 conditions more strongly on context, is more sensitive to ordering of events, and uses more unusual words, it is just as likely to produce repetitive and under-diverse text when using likelihood-maximizing decod- ing algorithms. 1 Introduction In 2018, large-scale neural models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) and OpenAI GPT (Radford et al., 2018) emerged as a dominant approach in NLP. By pretraining on massive amounts of unlabeled text (often or- ders of magnitude larger than the the target task’s labeled dataset), these models achieve state-of- the-art performance across a variety of Natural Language Understanding benchmarks. In partic- ular, the OpenAI GPT2 language model (Rad- ford et al., 2019) achieves state-of-the-art perfor- mance on several language modeling benchmarks, even in a zero-shot setting. While GPT2’s perfor- mance as a language model is undeniable, its per- formance as a text generator is much less clear. ∗equal contribution Though the model has generated certain impres- sive samples of text – such as a widely-circulated passage about Ovid’s Unicorn (Radford et al., 2019) – there has been no detailed study to for- malize these observations. In this work, we perform an in-depth study of the properties of text generated by GPT2-117 (the smallest version of GPT2) in the context of story generation. By comparing to a state-of-the- art, specialized-architecture neural story genera- tion model (Fan et al., 2018), we ask the follow- ing questions. In what ways does a large amount of open-domain pretraining data change the char- acteristics of generated text? In what ways does it make no difference? And is a task-specific archi- tecture necessary? For any probabilistic language model, the gen- erated text is strongly affected by the choice of de- coding algorithm – this is especially true for open- ended text generation tasks such as storytelling and chitchat dialogue (Kulikov et al., 2018; Holtz- man et al., 2019). Nevertheless, most natural lan- guage generation papers evaluate only one decod- ing algorithm – this …

Content truncated to 500 words