Facts2Story: Controlling Text Generation by Key Facts

Proceedings of the 28th International Conference on Computational Linguistics, pages 2329–2345 Barcelona, Spain (Online), December 8-13, 2020 2329 Facts2Story: Controlling Text Generation by Key Facts Eyal Orbach Bar Ilan University / Ramat Gan, Israel eyalorbach@gmail.com Yoav Goldberg Bar Ilan University / Ramat Gan, Israel Allen Institute for Artificial Intelligence yogo@cs.biu.ac.il Abstract Recent advancements in self-attention neural network architectures have raised the bar for open- ended text generation. Yet, while current methods are capable of producing a coherent text which is several hundred words long, attaining control over the content that is being generated—as well as evaluating it—are still open questions. We propose a controlled generation task which is based on expanding a sequence of facts, expressed in natural language, into a longer narrative. We introduce human-based evaluation metrics for this task, as well as a method for deriving a large training dataset. We evaluate three methods on this task, based on fine-tuning pre-trained models. We show that while auto-regressive, unidirectional Language Models such as GPT2 produce better fluency, they struggle to adhere to the requested facts. We propose a plan-and- cloze model (using fine-tuned XLNet) which produces competitive fluency while adhering to the requested content. 1 Introduction Story generation is a challenging task in natural language processing, which requires automated systems to produce creative, open-ended text that remains coherent, cohesive and preferably engaging: an ability humans are clearly capable of. Earlier attempts at harnessing neural techniques for story generation struggled with staying coherent for long periods of time, and proposed solutions in the form of two-stage processes which first generate skeletal structures and then expand them into longer texts (Harrison et al., 2017; Martin et al., 2018; Yao et al., 2018). Since then, advancements in self-attention architectures and large-scale training resulted in series of pre-trained language models (Radford et al., 2019; Zellers et al., 2019) that demonstrate an ability to remain on topic, while generating cohesive passages which are several hundreds of words long. While the generated text is almost indistinguishable from text produced by humans, the ability to control the contents of the text remains very limited and is still an active research topic (Zellers et al., 2019; Keskar et al., 2019; Ziegler et al., 2019; Dathathri et al., 2019). Attempts to control the content of the text focus on supplying a prefix (Radford et al., 2019; Keskar et al., 2019), a headline (Zellers et al., 2019) or a writing prompt (Fan et al., 2018; Ziegler et al., 2019), mechanisms that can influence the overall theme of the story, but do not provide much control over its content. Can we harness the power of large pre-trained models to generate coherent text, while allowing finer-grained controlled over the generated text? We suggest a task in which the content of the text is specified using a sequence off key facts expressed in natural language, which the story should follow, in order. An example of key facts 1 and a correspond- ing story appear in Figure 1. This formulation suggests natural evaluation …

Content truncated to 500 words