DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6408–6423 August 1–6, 2021. ©2021 Association for Computational Linguistics 6408 DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation Xinyu Hua1 Ashwin Sreevatsa2 Lu Wang2 1Khoury College of Computer Sciences, Northeastern University, Boston, MA 2Computer Science and Engineering, University of Michigan, Ann Arbor, MI 1hua.x@northeastern.edu 2{asreeva, wangluxy}@umich.edu Abstract We study the task of long-form opinion text generation, which faces at least two distinct challenges. First, existing neural generation models fall short of coherence, thus requir- ing efficient content planning. Second, diverse types of information are needed to guide the generator to cover both subjective and objec- tive content. To this end, we propose DY- PLOC, a generation framework that conducts dynamic planning of content while generating the output based on a novel design of mixed language models. To enrich the generation with diverse content, we further propose to use large pre-trained models to predict relevant concepts and to generate claims. We experi- ment with two challenging tasks on newly col- lected datasets: (1) argument generation with Reddit ChangeMyView, and (2) writing arti- cles using New York Times’ Opinion section. Automatic evaluation shows that our model significantly outperforms competitive compar- isons. Human judges further confirm that our generations are more coherent with richer con- tent. 1 Introduction Opinion articles serve as an important media to convey the authors’ values, beliefs, and stances on important societal issues. Automatically generat- ing long-form opinion articles has the potential of facilitating various tasks, such as essay writing and speech drafting, and it is the focus of this work. Though opinion generation has been investigated for constructing arguments (Hua and Wang, 2018), writing reviews (Ni and McAuley, 2018), and pro- ducing emotional dialogue responses (Song et al., 2019), those outputs are relatively short. While im- pressive progress in generation has been achieved by using large pre-trained Transformers (Radford et al., 2019; Lewis et al., 2020a), directly adopting United_States, Intelligence knowledge, attack America was never prepared and had a bad intelligence system. President_of_the_U.S., Bill_Clinton, 9/11_attacks make, happen, mistake, administration George_W._Bush, 9/11_attacks, Iraq existence 1) -> 2) -> 3) -> Figure 1: Sample counter-argument on Reddit Change- MyView. Our generator considers an input containing (1) a title and (2) an unordered set of content items. Each content item consists of elements of an entity set [ENT], a concept set [CON], and an optional one- sentence claim [CLAIM]. Each output token is gener- ated by conditioning on all content items, and the best aligned ones (learned by our model) are highlighted in corresponding colors. We also underline words that re- flect the input concepts and entities. them for long-form opinion text generation poses distinct challenges. First, large models still fall short of producing coherent text due to the lack of efficient content control and planning (Ko and Li, 2020; Wu et al., 2020; Tan et al., 2021). A common solution is to use …

Content truncated to 500 words