The Discussion Tracker Corpus of Collaborative Argumentation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1033–1043 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1033 The Discussion Tracker Corpus of Collaborative Argumentation Christopher Olshefski, Luca Lugini, Ravneet Singh, Diane Litman, Amanda Godley University of Pittsburgh {cao48, lul32, ras306, dlitman, agodley}@pitt.edu Abstract Although Natural Language Processing (NLP) research on argument mining has advanced considerably in recent years, most studies draw on corpora of asynchronous and written texts, often produced by individuals. Few published corpora of synchronous, multi-party argumentation are available. The Discussion Tracker corpus, collected in American high school English classes, is an annotated dataset of transcripts of spoken, multi-party argumentation. The corpus consists of 29 multi-party discussions of English literature transcribed from 985 minutes of audio. The transcripts were annotated for three dimensions of collaborative argumentation: argument moves (claims, evidence, and explanations), specificity (low, medium, high) and collaboration (e.g., extensions of and disagreements about others’ ideas). In addition to providing descriptive statistics on the corpus, we provide performance benchmarks and associated code for predicting each dimension separately, illustrate the use of the multiple annotations in the corpus to improve performance via multi-task learning, and finally discuss other ways the corpus might be used to further NLP research. Keywords: Corpus, Discourse, Text mining 1. Introduction Natural Language Processing (NLP) research has advanced considerably in recent years, developing reliable predic- tors for argumentation (Mirkin et al., 2018; Lippi and Torroni, 2016; Aharoni et al., 2014; Biran and Rambow, 2011a; Carlile et al., 2018; Habernal et al., 2014; Park and Cardie, 2018; Stab and Gurevych, 2014; Stab and Gurevych, 2017a), specificity (Li and Nenkova, 2015; Li et al., 2016; Louis and Nenkova, 2012; Gao et al., 2019), and collaboration (Richey et al., 2016). The majority of cor- pora informing these developments are made up of written texts (e.g., documents, asynchronous online discussions) extracted from the web and often written by individuals. Additionally, annotations in these corpora typically focus on one linguistic dimension at a time (e.g., specificity or dimensions of argumentation). As such, few published cor- pora both (1) focus on synchronous multi-party argumenta- tion and (2) include multiple simultaneous annotations. To address the lack of multi-party synchronous argu- mentation corpora that include multiple simultaneous an- notations, we are releasing the Discussion Tracker cor- pus, which includes transcriptions of 29 multi-party syn- chronous argument-based dialogues collected in high school English classes and annotated for three simultaneous but distinct discourse dimensions (argumentation, speci- ficity and collaboration). In addition to providing descrip- tive statistics on the corpus, we provide code and bench- mark performance figures for predicting each of the three annotated dimensions, illustrating the challenging nature of this corpus. We then illustrate the benefits the corpus has already afforded NLP algorithms for improving argument mining with multi-task learning and discuss other potential uses of the corpus for further NLP research. 2. Related Work The development of argument mining tasks has been heav- ily informed by corpora of written texts extracted mostly …

Content truncated to 500 words