MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2831–2845, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2831 MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models Peng Xu‡∗, Mostofa Patwary§, Mohammad Shoeybi§, Raul Puri§, Pascale Fung‡, Anima Anandkumar§, Bryan Catanzaro§ ‡The Hong Kong University of Science and Technology, §NVIDIA peng.xu@connect.ust.hk, mpatwary@nvidia.com Abstract Existing pre-trained large language models have shown unparalleled generative capabili- ties. However, they are not controllable. In this paper, we propose MEGATRON-CNTRL, a novel framework that uses large-scale lan- guage models and adds control to text gener- ation by incorporating an external knowledge base. Our framework consists of a keyword predictor, a knowledge retriever, a contextual knowledge ranker, and a conditional text gen- erator. As we do not have access to ground- truth supervision for the knowledge ranker, we make use of weak supervision from sentence embedding. The empirical results show that our model generates more fluent, consistent, and coherent stories with less repetition and higher diversity compared to prior work on the ROC story dataset. We showcase the controlla- bility of our model by replacing the keywords used to generate stories and re-running the generation process. Human evaluation results show that 77.5% of these stories are success- fully controlled by the new keywords. Further- more, by scaling our model from 124 million to 8.3 billion parameters we demonstrate that larger models improve both the quality of gen- eration (from 74.5% to 93.0% for consistency) and controllability (from 77.5% to 91.5%). 1 Introduction Text generation has recently attracted significant attention from the research community as large pre- trained language models, such as GPT-2 (Radford et al., 2018, 2019) demonstrated promising results for generating long, grammatically correct, and flu- ent text. Finetuning these models has shown signif- icant improvements in downstream tasks, such as persona chat (Wolf et al., 2019). However, one non- negligible drawback of these large models is the ∗This work was done during the internship of Peng Xu at NVIDIA. Corresponding authors: Peng Xu, Mostofa Patwary. Context [FEMALE] was on a long road trip . Guan et al. (2020) she had just driven into town . she had been driving for hours . she was so tired she decided to take a nap . when she woke up , she still had n’t slept well . MEGATRON- CNTRL-124M keyword: driving she was on her way to see a strange country. she had never seen anything like that before. finally she saw a small town that was so small. it was a great little town with a nice park ! MEGATRON- CNTRL-8B keyword: driving she was driving in the middle of [PLACE]. all of a sudden the tire pressure light came on. she got out to check her tire. it was flat so she used the roadside assistance. MEGATRON- CNTRL-8B-ANT keyword: attract she really wanted to see a few attractions. the first one she saw was a giant water park. it was amazing. it ended up being a fun …

Content truncated to 500 words