A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation

A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation Jian Guan1 Fei Huang1 Zhihao Zhao2 Xiaoyan Zhu1 Minlie Huang1∗ 1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China 2School of Software, Beihang University, Beijing, China 1Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems 1Beijing National Research Center for Information Science and Technology j-guan19@mails.tsinghua.edu.cn,f-huang18@mails.tsinghua.edu.cn, extsuioku@gmail.com, zxy-dcs@tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Story generation, namely, generating a rea- sonable story from a leading context, is an im- portant but challenging task. In spite of the success in modeling fluency and local coher- ence, existing neural language generation models (e.g., GPT-2) still suffer from repe- tition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of asso- ciating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper tem- poral order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from exter- nal knowledge bases to generate reasonable stories. To further capture the causal and tem- poral dependencies between the sentences in a reasonable story, we use multi-task learning, which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evalua- tion shows that our model can generate more reasonable stories than state-of-the-art base- lines, particularly in terms of logic and global coherence. 1 Introduction Story generation is a strong indicator of machine understanding of natural language. It is often approached as selecting a sequence of events to form a story with a reasonable logic or plot. Although existing generative models (Roemmele, 2016; Fan et al., 2018; Fan et al., 2019) can generate stories with good local coherence, they ∗Corresponding author: Minlie Huang. are still struggling to plan a coherent plot and maintain a reasonable event sequence throughout the story, or they are often biased towards generating a limited set of stories with generic plots (See et al., 2019) (e.g., I have a great time), even when using the powerful generative model OpenAI’s GPT-2 (Radford et al., 2019), as shown in Table 1. Pretrained GPT-2 has been shown to cap- ture useful semantic and syntactic features (Alt et al., 2019), as demonstrated by state-of-the- art performance on some generation tasks such as machine translation and text summarization (Radford et al., 2019). However, compared with such tasks whose source inputs have contained sufficient information to generate desired tar- get texts, story generation is a typical open- ended generation task, where only very limited information is given in the input. As shown in this paper, we observe some severe issues when applying GPT-2 to generate reasonable stories, particularly commonsense stories from a limited beginning. These issues include repetition, logic conflicts, and lack of long-range coherence (See et al., 2019; Holtzman et al., 2019), as exemplified in Table 1. Specifically, although GPT-2 performs reasonably well at generating some related concepts to bus (e.g., driver, and the probable destinations home or party), it completely ignores the …

Content truncated to 500 words