Weighted Krippendorff’s alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 550–559, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Weighted Krippendorff's alpha is a more reliable metrics for multi- coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation Jean-Yves Antoine Université François Rabelais de Tours, LI (EA 6300) Blois, France Jean-Yves.Antoine@univ-tours.fr Jeanne Villaneau Université Européenne de Bretagne, IRISA Lorient, France Jeanne.Villaneau@univ-ubs.fr Anaïs Lefeuvre Université François Rabelais de Tours, LI (EA 6300) Blois, France anais.lefeuvre@univ-tours.fr Abstract The question of data reliability is of first im- portance to assess the quality of manually an- notated corpora. Although Cohen’s κ is the prevailing reliability measure used in NLP, al- ternative statistics have been proposed. This paper presents an experimental study with four measures (Cohen’s κ, Scott’s π, binary and weighted Krippendorff’s α) on three tasks: emotion, opinion and coreference annotation. The reported studies investigate the factors of influence (annotator bias, category prevalence, number of coders, number of categories) that should affect reliability estimation. Results show that the use of a weighted measure re- stricts this influence on ordinal annotations. They suggest that weighted α is the most reli- able metrics for such an annotation scheme. 1 Introduction The newly intensive use of machine learning techniques as well as the need of evaluation data has led Natural Language Processing (NLP) to develop large annotated corpora. The interest for such enriched language resources has reached domains (semantics, pragmatics, affective com- puting) where the annotation process is highly affected by the coders subjectivity. The reliabil- ity of the resulting annotations must be trusted by measures that assess the inter-coders agreement. While medecine, psychology, and more gener- ally content analysis, have considered for years the issue of data reliability, NLP has only inves- tigated this question from the mid 1990s. The influential work of Carletta (1996) has led the κ statistic (Cohen, 1960) to become the prevailing standard for measuring the reliability of corpus annotation. Many studies have however ques- tioned the limitations of the κ statistic and have proposed alternative measures of reliability. Krippendorff claims that “popularity of κ not- withstanding, Cohen’s κ is simply unsuitable as a measure of the reliability of data” in a paper presenting his α coefficient (Krippendorff, 2008). Except for some rare but noticeable studies (Arstein and Poesio, 2005), most of these critical works restrict to theoretical issues about chance agreement estimation or limitations due to vari- ous statistical biases (Arstein and Poesio, 2008). On the opposite, this paper investigates experi- mentally these questions on three different tasks: emotion, opinion and coreference annotation. Four measures of reliability will be considered: Cohen’s κ (Cohen, 1960), Scott’s π (Scott, 1955) and two measures of Krippendorff’s α (Krippen- dorff, 2004) with different distance. Section 2 gives a comprehensive presentation of these metrics. Section 3 details the potential methodological biases that should affect the reli- ability estimation. In section 4, we explain the methodology we followed for this study. Lastly, experimental results are presented in …

Content truncated to 500 words