A Corpus for Modeling User and Language Effects in Argumentation on Online Debating

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 602–607 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 602 A Corpus for Modeling User and Language Effects in Argumentation on Online Debating Esin Durmus Cornell University ed459@cornell.edu Claire Cardie Cornell University cardie@cs.cornell.edu Abstract Existing argumentation datasets have suc- ceeded in allowing researchers to develop computational methods for analyzing the con- tent, structure and linguistic features of argu- mentative text. They have been much less successful in fostering studies of the effect of “user” traits — characteristics and beliefs of the participants — on the debate/argument outcome as this type of user information is generally not available. This paper presents a dataset of 78, 376 debates generated over a 10-year period along with surprisingly com- prehensive participant profiles. We also com- plete an example study using the dataset to an- alyze the effect of selected user traits on the debate outcome in comparison to the linguistic features typically employed in studies of this kind. 1 Introduction Previous work from Natural Language Processing (NLP) and Computational Social Science (CSS) that studies argumentative text and its persuasive effects has mainly focused on identifying the con- tent and structure of an argument (e.g. Feng and Hirst (2011)) and the linguistic features that are in- dicative of effective argumentation strategies (e.g. Tan et al. (2016)). The effectiveness of an argu- ment, however, cannot be determined solely by its textual content; rather, it is important to consider characteristics of the reader, listener or partici- pants in the debate or discussion. Does the reader already agree with the argument’s stance? Is she predisposed to changing her mind on the particu- lar topic of the debate? Is the style of the argu- ment appropriate for the individual? To date, ex- isting argumentation datasets have permitted only limited assessment of such “user” traits because information on the background of users is gen- erally unavailable. In this paper, we present a dataset of 78, 376 debates from October of 2007 until November of 2017 drawn from debate.org along with quite comprehensive user profile infor- mation — for debate participants as well as users voting on the debate quality and outcome. Back- ground information on users includes demograph- ics (e.g. education, income, religion) and stance on a variety of controversial debate topics as well as a record of user activity on the debate platform (e.g. debates won and lost). We view this new dataset as a resource that affords the NLP and CSS commu- nities the opportunity to understand the effect of audience characteristics on the efficacy of differ- ent debating and persuasion strategies as well as to model changes in user’s opinions and activities on a debate platform over time. (To date, part of our debate.org dataset has been used in one such study to understand the effect of prior beliefs in persua- sion1 (Durmus and Cardie, 2018). Here, we focus on the properties of the dataset itself and study a …

Content truncated to 500 words