Measuring Online Debaters’ Persuasive Skill from Text over Time

Measuring Online Debaters’ Persuasive Skill from Text over Time Kelvin Luu1 Chenhao Tan2 Noah A. Smith1,3 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Department of Computer Science, University of Colorado Boulder 3Allen Institute for Artificial Intelligence {kellu,nasmith}@cs.washington.edu chenhao.tan@colorado.edu Abstract Online debates allow people to express their persuasive abilities and provide exciting oppor- tunities for understanding persuasion. Prior studies have focused on studying persuasion in debate content, but without accounting for each debater’s history or exploring the progression of a debater’s persuasive ability. We study debater skill by modeling how participants progress over time in a collection of debates from Debate.org. We build on a widely used model of skill in two-player games and augment it with linguistic features of a debater’s content. We show that online debaters’ skill levels do tend to improve over time. Incorporating linguistic profiles leads to more robust skill estimation than winning records alone. Notably, we find that an interaction feature combining uncertainty cues (hedging) with terms strongly associated with either side of a particular debate (fightin’ words) is more predictive than either feature on its own, indicating the importance of fine- grained linguistic features. 1 Introduction Persuasion is an important skill with prevalent use. Nearly every kind of social encounter, from formal political debates to casual conversations, can include attempts to convince others. What linguistic phenomena are associated with higher levels of persuasive skill? How do people develop this skill over time? Online debate communities offer an opportunity to investigate these questions. These communities feature users who participate in multiple debates over their period of engage- ment. Such debates involve two parties who willingly and formally present divergent opinions before an audience. Unlike other media of persuasion, such as letters to politicians, there is a clear signal, a win or loss, indicating whether or not a debater was successful against the adversary. This work aims to quantify the skill level of each debater in an online community and also investigates what factors contribute to expertise. Although persuasion has generated interest in the natural language processing community, most researchers have not tried to quantify the persuasiveness of a particular speaker. Instead, they estimate how persuasive a text is, using linguistic features such as the author’s choice of wording or how they interact with the audience (Tan et al., 2014, 2016; Althoff et al., 2014; Danescu-Niculescu-Mizil et al., 2012). Previous research has also established that debaters’ content and interactions both contribute to the success of the persuader (Tan et al., 2016; Zhang et al., 2016; Wang et al., 2017). These works have found textual factors that contribute to a debater’s success, but they do not emphasize the role of the individual debater. There has been some recent work on studying individual debaters. Durmus and Cardie (2019) analyze users and find that a user’s success and improvement depend on their social network features. We take another approach by estimating each user’s skill level in each debate they par- ticipate in by considering their …

Content truncated to 500 words