Machine-Aided Annotation for Fine-Grained Proposition Types in Argumentation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1008–1018 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1008 Machine-Aided Annotation for Fine-Grained Proposition Types in Argumentation Yohan Jo1, Elijah Mayfield1, Chris Reed2, Eduard Hovy1 1Language Technologies Institute, Carnegie Mellon University, USA 2Centre for Argument Technology, University of Dundee, UK yohanj@cs.cmu.edu, elijah@cmu.edu, c.a.reed@dundee.ac.uk, hovy@cmu.edu Abstract We introduce a corpus of the 2016 U.S. presidential debates and commentary, containing 4,648 argumentative propositions annotated with fine-grained proposition types. Modern machine learning pipelines for analyzing argument have difficulty distinguishing between types of propositions based on their factuality, rhetorical positioning, and speaker commitment. Inability to properly account for these facets leaves such systems inaccurate in understanding of fine-grained proposition types. In this paper, we demonstrate an approach to annotating for four complex proposition types, namely normative claims, desires, future possibility, and reported speech. We develop a hybrid machine learn- ing and human workflow for annotation that allows for efficient and reliable annotation of complex linguistic phenomena, and demonstrate with preliminary analysis of rhetorical strategies and structure in presidential debates. This new dataset and method can support techni- cal researchers seeking more nuanced representations of argument, as well as argumentation theorists developing new quantitative analyses. Keywords: argumentation theory, proposition types, imbalanced annotation tasks, hybrid annotation systems 1. Introduction Argument mining is a broad field of computational linguis- tics that seeks to identify the structure of written and spoken argument and extract meaningful content based on that un- derstanding. But as the domains that we can tackle with NLP grow more diverse, and expand from newswire text to social media and real-world dialogue, we are reaching an inflection point. These domains are not characterized solely by objective statements with clean reporting of facts and details; opinion, hedging, and reported speech are common- place. In recent years, researchers have found that argument mining pipelines struggle to identify factual content and disambiguate it from fiction, lies, or mere hypotheticals in real-world data (Feng et al., 2012; Thorne et al., 2018). In today’s politically charged atmosphere, this poses a chal- lenge for developers of systems like fake news detectors and recommender systems: when algorithmic systems cannot even reliably detect the presence or assertion of facts in statements, how can they address the ethical challenges of deployed machine learning systems at scale (Leidner and Plachouras, 2017; Gonen and Goldberg, 2019)? This paper introduces new resources for understanding propositions that appear in speech and text, based on the 2016 U.S. presidential debates. We define a fine-grained, four-dimensional annotation schema for how propositions are introduced rhetorically in debates: namely, normative statements, statements of desire, statements about future possibility, and reported speech. These proposition types are tied closely to practical reasoning, causal reasoning, and authority claims in argumentation schemes (Walton et al., 2008) and represent varying levels of speaker commitment to individual statements (Lasersohn, 2009). While these definitions are tractable for reliable human an- notators, we find that occurrences in running text are rare …

Content truncated to 500 words