Increasing Argument Annotation Reproducibility by Using Inter-annotator Agreement to Improve Guidelines

Increasing Argument Annotation Reproducibility by Using Inter-annotator Agreement to Improve Guidelines Milagro Teruel1, Cristian Cardellino1, Fernando Cardellino1, Laura Alonso Alemany1, Serena Villata2 1Universidad Nacional de C´ordoba, Argentina 2Universit´e Cˆote d’Azur, CNRS, Inria, I3S, France mteruel@unc.edu.ar, ccardellino@unc.edu.ar, fndcardellino@gmail.com, lauraalonsoalemany@unc.edu.ar, serena.villata@inria.fr Abstract In this abstract we present a methodology to improve Argument annotation guidelines by exploiting inter-annotator agreement measures. After a first stage of the annotation effort, we have detected problematic issues via an analysis of inter-annotator agreement. We have detected ill-defined concepts, which we have addressed by redefining high-level annotation goals. For other concepts, that are well-delimited but complex, the annotation protocol has been extended and detailed. Moreover, as can be expected, we show that distinctions where human annotators have less agreement are also those where automatic analyzers perform worse. Thus, the reproducibility of results of Argument Mining systems can be addressed by improving inter-annotator agreement in the training material. Following this methodology, we are enhancing a corpus annotated with argumentation, available at https://github.com/ PLN-FaMAF/ArgumentMiningECHR together with guidelines and analyses of agreement. These analyses can be used to filter performance figures of automated systems, with lower penalties for cases where human annotators agree less. Keywords: Argument Mining, Annotation Guidelines, Reproducibility 1. Introduction and Motivation Argument Mining tackles a very complex phenomenon, in- volving several levels of human communication and cog- nition. Due to this complexity, data-driven approaches re- quire a huge amount of data to properly characterize the phenomena and find patterns that can be exploited by an automatic analyzer. However, only small annotated cor- pora are available, and moreover they cannot be used in combination because they are based on different theoretical frameworks or cover different genres. In this abstract we present work in progress in building a corpus annotated with arguments. As inherent part of this work, we are applying a methodology for early detection of ill-defined annotation concepts. We detect those by inspect- ing annotated texts for sources of disagreement between an- notators, and redefining the annotation scheme so that these disagreements are minimized. This has been a successful approach to improve guidelines, as in (Hovy et al., 2006). In preliminary explorations, we have found that agreement- driven modifications in the annotation scheme improve an automatic analyzer. Our final objective is to find an anno- tation scheme that is a tradeoff between theoretically based concepts, application needs, stability of human annotation and performance of automatic analyzers. 2. Annotated Corpus 2.1. ECHR Judgments Four human annotators have annotated 7 judgments from the European Court of Human Rights (ECHR) in English, obtained from the Court website1, totaling 28,000 words. Approximately half of the words were annotated as belong- ing to an argument component, as can be seen in Figure 1. One of the judgments was annotated by all 4 annota- tors and discussed collectively as training. In this anno- tation, agreement between judges was never lower than 1hudoc.echr.coe.int Figure 1: Proportion of component labels in the corpus. κ = .54. Then, two pairs of judges annotated two judgments independently, we …

Content truncated to 500 words