Changing the Mind of Transformers for Topically-Controllable Language Generation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2601–2611 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2601 Changing the Mind of Transformers for Topically-Controllable Language Generation Haw-Shiuan Chang Jiaming Yuan Mohit Iyyer Andrew McCallum CICS, University of Massachusetts Amherst hschang@cs.umass.edu,jiamingyuan@umass.edu, {mccallum,miyyer}@cs.umass.edu Abstract Large Transformer-based language models can aid human authors by suggesting plausi- ble continuations of text written so far. How- ever, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this lim- itation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the gener- ation. Our framework consists of two compo- nents: (1) a method that produces a set of can- didate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics. The training of both compo- nents is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework of- ten generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers. 1 Introduction Recently, Transformer-based language models (LMs) have achieved impressive performance in language generation tasks (Radford et al., 2019; Dai et al., 2019) such as open-domain story genera- tion (See et al., 2019a). When writing with the LM, users often desire an intuitive and effective way to control what a LM is going to generate (Keskar et al., 2019). To address this need, interactive writ- ing assistants provide options to reveal possible developments of the story and generate continua- tions guided by the user-selected options. Interactive writing assistants have wide applica- tions in creative writing (Roemmele and Gordon, 2015; Clark et al., 2018; Akoury et al., 2020), ed- ucation (Luo et al., 2015), and gaming (Walton, 2020). Nevertheless, the existing systems’ options usually do not provide fine-grained control and/or Step 2: Might say these topics Step 1: Let’s see what language models would say Step 3: Please talk more about these topics 1 book books novels 2 Essays Perspectives Perspective 3 University faculty undergraduate 4 Reid Sen. McConnell 5 humanity life spirituality 6 2011 2010 2009 7 know sure want 8 insistence disdain dismissive 9 election elections Democratic 10 U.S. States United Input Prompt: “Barack Obama writes a new book” Output Continuation: “: The Future of a Democratic Election. The book tells the story of the 2008 election.” Transformer -based Language Models User Step 4: Let me try. What does this continuation sound? Figure 1: Given an input prompt, the Transformer- based LM provides K = 10 topics that might be men- tioned next and each topic is represented by M = 3 words. The user could guide the generation process by choosing a subset of topics. require substantial human labor. In some prior work (Keskar et al., 2019; Tu et al., 2019), …

Content truncated to 500 words