Anchoring and Agreement in Syntactic Annotations

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2215–2224, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Anchoring and Agreement in Syntactic Annotations Yevgeni Berzak CSAIL MIT berzak@mit.edu Yan Huang Language Technology Lab DTAL Cambridge University yh358@cam.ac.uk Andrei Barbu CSAIL MIT andrei@0xab.com Anna Korhonen Language Technology Lab DTAL Cambridge University alk23@cam.ac.uk Boris Katz CSAIL MIT boris@mit.edu Abstract We present a study on two key character- istics of human syntactic annotations: an- choring and agreement. Anchoring is a well known cognitive bias in human decision mak- ing, where judgments are drawn towards pre- existing values. We study the influence of anchoring on a standard approach to creation of syntactic resources where syntactic annota- tions are obtained via human editing of tagger and parser output. Our experiments demon- strate a clear anchoring effect and reveal un- wanted consequences, including overestima- tion of parsing performance and lower qual- ity of annotations in comparison with human- based annotations. Using sentences from the Penn Treebank WSJ, we also report systemat- ically obtained inter-annotator agreement es- timates for English dependency parsing. Our agreement results control for parser bias, and are consequential in that they are on par with state of the art parsing performance for En- glish newswire. We discuss the impact of our findings on strategies for future annotation ef- forts and parser evaluations.1 1 Introduction Research in NLP relies heavily on the availability of human annotations for various linguistic prediction tasks. Such resources are commonly treated as de facto “gold standards” and are used for both training 1The experimental data in this study will be made publicly available. and evaluation of algorithms for automatic annota- tion. At the same time, human agreement on these annotations provides an indicator for the difficulty of the task, and can be instrumental for estimating upper limits for the performance obtainable by com- putational methods. Linguistic gold standards are often constructed using pre-existing annotations, generated by auto- matic tools. The output of such tools is then man- ually corrected by human annotators to produce the gold standard. The justification for this annotation methodology was first introduced in a set of exper- iments on POS tag annotation conducted as part of the Penn Treebank project (Marcus et al., 1993). In this study, the authors concluded that tagger-based annotations are not only much faster to obtain, but also more consistent and of higher quality compared to annotations from scratch. Following the Penn Treebank, syntactic annotation projects for various languages, including German (Brants et al., 2002), French (Abeill´e et al., 2003), Arabic (Maamouri et al., 2004) and many others, were annotated us- ing automatic tools as a starting point. Despite the widespread use of this annotation pipeline, there is, to our knowledge, little prior work on syntactic an- notation quality and on the reliability of system eval- uations on such data. In this work, we present a systematic study of the influence of automatic tool output on characteristics of annotations created for NLP purposes. Our …

Content truncated to 500 words