A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd

Proceedings of NAACL-HLT 2019, pages 1790–1796 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1790 A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd Tristan Miller*† and Maria Sukhareva‡ and Iryna Gurevych*† *Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universität Darmstadt https://www.ukp.tu-darmstadt.de/ †Research Training Group AIPHES Department of Computer Science, Technische Universität Darmstadt https://www.aiphes.tu-darmstadt.de/ ‡NLP Lab BMW Group, Munich Maria.Sukhareva@bmwgroup.com Abstract The study of argumentation and the develop- ment of argument mining tools depends on the availability of annotated data, which is chal- lenging to obtain in sufficient quantity and qual- ity. We present a method that breaks down a popular but relatively complex discourse-level argument annotation scheme into a simpler, it- erative procedure that can be applied even by untrained annotators. We apply this method in a crowdsourcing setup and report on the relia- bility of the annotations obtained. The source code for a tool implementing our annotation method, as well as the sample data we ob- tained (4909 gold-standard annotations across 982 documents), are freely released to the re- search community. These are intended to serve the needs of qualitative research into argumen- tation, as well as of data-driven approaches to argument mining. 1 Introduction Empirical study of argumentation requires exam- ples drawn from authentic, human-authored text. Likewise, the applications of computational argu- mentation, such as argument mining, can require significant amounts of argument-annotated data to achieve reasonable performance. However, this data can be challenging to obtain in sufficient quan- tity and quality, particularly for discourse-level argumentation. This is because discourse-level annotation schemes are necessarily complex with respect to discrimination and delimitation (i.e., the variety of markable elements in the text and how to define their boundaries), expressiveness (i.e., the need to tag relationships between annotated elements), and context weighting (i.e., the amount of context around markable units that needs to be considered) (Fort et al., 2012). Successfully apply- ing such schemes typically requires expensive and laborious work by expert-trained annotators. In this paper, we present a method that facili- tates the application of one such discourse-level argument annotation scheme (Stab and Gurevych, 2014). This scheme has been widely cited and used in argumentation studies (e.g., Lippi and Torroni, 2015; Persing and Ng, 2015; Nguyen and Litman, 2015; Persing and Ng, 2016; Ghosh et al., 2016; Eger et al., 2017; Nguyen and Litman, 2018), and while it is fairly coarse-grained, it is expensive to apply to new texts. Our method breaks down the annotation process into incremental, intuitive steps, each focusing on a small portion of the overall annotation scheme. We apply this method in a crowdsourcing setup with annotators who receive no training other than a brief set of annotation guidelines, as well as in a more traditional setup with extensively trained local annotators. We find that agreement between the two groups increases sublinearly with the number of crowd annotators, achieving up to αU = 0.52 when using ten crowd workers. We release not only our sample …

Content truncated to 500 words