The workweek is the best time to start a family – A Study of GPT-2 Based Claim Generation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 528–544 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 528 The workweek is the best time to start a family – A Study of GPT-2 Based Claim Generation Shai Gretz∗, Yonatan Bilu∗, Edo Cohen-Karlik and Noam Slonim IBM Research {avishaig,yonatanb,noams}@il.ibm.com {edo.cohen}@ibm.com Abstract Argument generation is a challenging task whose research is timely considering its po- tential impact on social media and the dissem- ination of information. Here we suggest a pipeline based on GPT-2 for generating coher- ent claims, and explore the types of claims that it produces, and their veracity, using an array of manual and automatic assessments. In ad- dition, we explore the interplay between this task and the task of Claim Retrieval, showing how they can complement one another. 1 Introduction Argument Mining had traditionally focused on the detection and retrieval of arguments, and the clas- sification of their types and of the relations among them. Recently, there has been growing interest in argument synthesis. Here we suggest a pipeline for addressing this task relying on the GPT-2 lan- guage model (Radford et al., 2019), examine how it can be enhanced to provide better arguments, and analyze the types of arguments being produced. Specifically, we are interested in Claim Generation, where the input is a debate topic, phrased as a pro- posed policy, and the output is a concise assertion with a clear stance on this topic. We start by fine-tuning GPT-2 on a collection of topics and associated claims. Since several such datasets are available, we examine which of them tend to yield better claims, and observe that merg- ing all such sources together does not necessarily yield better results. In addition, we explore two ways in which context can be added to the genera- tion process, beyond providing the topic itself: (i) framing the topic with the first sentence from its corresponding Wikipedia page; and (ii) framing the claim by directing it to consider a specific aspect. We find that the former can improve the generated ∗These authors equally contributed to this work. output, but the latter does not – at least in the way it is done here. Following Bilu and Slonim (2016), we also examine a post-generation ranking step that aims to select the correctly generated claims. We find that existing Claim Detection tools can serve as a filter to significantly enhance generation quality. Our evaluation incorporates automatic measures and manual labeling. Specifically, we introduce an annotation task aiming to assess the plausibility of generated claims, i.e., to what degree is it plausible that a human will make it. We report results on a test set of 96 topics, demonstrating the validity of our approach to topics not seen in training or development. In addition, we manually annotate the generated claims for whether they are factual claims, or opinion based, and further aim to assess whether the former represent true facts. Finally, we observe that manually labeled datasets used to fine-tune …

Content truncated to 500 words