A chance-corrected measure of inter-annotator agreement for syntax

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 934–944, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics A chance-corrected measure of inter-annotator agreement for syntax Arne Skjærholt Language technology group, dept. of informatics University of Oslo arnskj@ifi.uio.no Abstract Following the works of Carletta (1996) and Artstein and Poesio (2008), there is an increasing consensus within the field that in order to properly gauge the reliability of an annotation effort, chance-corrected measures of inter-annotator agreement should be used. With this in mind, it is striking that virtually all evaluations of syntactic annotation efforts use uncor- rected parser evaluation metrics such as bracket F1 (for phrase structure) and ac- curacy scores (for dependencies). In this work we present a chance-corrected metric based on Krippendorff’s α, adapted to the structure of syntactic annotations and applicable both to phrase structure and dependency annotation without any modifications. To evaluate our metric we first present a number of synthetic experi- ments to better control the sources of noise and gauge the metric’s responses, before finally contrasting the behaviour of our chance-corrected metric with that of un- corrected parser evaluation metrics on real corpora.1 1 Introduction It is a truth universally acknowledged that an an- notation task in good standing be in possession of a measure of inter-annotator agreement (IAA). However, no such measure is in widespread use for the task of syntactic annotation. This is due to a mismatch between the formulation of the agree- ment measures, which assumes that the annota- tions have no or relatively little internal structure, 1The code used to produce the data in this paper, and some of the datasets used, are available to download at https://github.com/arnsholt/syn-agreement/ and syntactic annotation where structure is the en- tire point of the annotation. For this reason efforts to gauge the quality of syntactic annotation are hampered by the need to fall back to simple ac- curacy measures. As shown in Artstein and Poesio (2008), such measures are biased in favour of an- notation schemes with fewer categories and do not account for skewed distributions between classes, which can give high observed agreement, even if the annotations are inconsistent. In this article we propose a family of chance- corrected measures of agreement, applicable to both dependency- and constituency-based syntac- tic annotation, based on Krippendorff’s α and tree edit distance. First we give an overview of tradi- tional agreement measures and why they are insuf- ficient for syntax, before presenting our proposed metrics. Next, we present a number of synthetic experiments performed in order to find the best distance function for this kind of annotation; fi- nally we contrast our new metric and simple accu- racy scores as applied to real-world corpora before concluding and presenting some potential avenues for future work. 1.1 Previous work The definitive reference for agreement measures in computational linguistics is Artstein and Poe- sio (2008), who argue forcefully in favour of the use of chance-corrected measures of agreement over simple accuracy measures. However, …

Content truncated to 500 words