A Cognitive Cost Model of Annotations Based on Eye-Tracking Data

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1158–1167, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics A Cognitive Cost Model of Annotations Based on Eye-Tracking Data Katrin Tomanek Language & Information Engineering (JULIE) Lab Universit¨at Jena Jena, Germany Udo Hahn Language & Information Engineering (JULIE) Lab Universit¨at Jena Jena, Germany Steffen Lohmann Dept. of Computer Science & Applied Cognitive Science Universit¨at Duisburg-Essen Duisburg, Germany J¨urgen Ziegler Dept. of Computer Science & Applied Cognitive Science Universit¨at Duisburg-Essen Duisburg, Germany Abstract We report on an experiment to track com- plex decision points in linguistic meta- data annotation where the decision behav- ior of annotators is observed with an eye- tracking device. As experimental con- ditions we investigate different forms of textual context and linguistic complexity classes relative to syntax and semantics. Our data renders evidence that annotation performance depends on the semantic and syntactic complexity of the decision points and, more interestingly, indicates that full- scale context is mostly negligible – with the exception of semantic high-complexity cases. We then induce from this obser- vational data a cognitively grounded cost model of linguistic meta-data annotations and compare it with existing non-cognitive models. Our data reveals that the cogni- tively founded model explains annotation costs (expressed in annotation time) more adequately than non-cognitive ones. 1 Introduction Today’s NLP systems, in particular those rely- ing on supervised ML approaches, are meta-data greedy. Accordingly, in the past years, we have witnessed a massive quantitative growth of anno- tated corpora. They differ in terms of the nat- ural languages and domains being covered, the types of linguistic meta-data being solicited, and the text genres being served. We have seen large- scale efforts in syntactic and semantic annotations in the past related to POS tagging and parsing, on the one hand, and named entities and rela- tions (propositions), on the other hand. More re- cently, we are dealing with even more challeng- ing issues such as subjective language, a large variety of co-reference and (e.g., RST-style) text structure phenomena, Since the NLP community is further extending their work into these more and more sophisticated semantic and pragmatic analyt- ics, there seems to be no end in sight for increas- ingly complex and diverse annotation tasks. Yet, producing annotations is pretty expensive. So the question comes up, how we can rationally manage these investments so that annotation cam- paigns are economically doable without loss in an- notation quality. The economics of annotations are at the core of Active Learning (AL) where those linguistic samples are focused on in the entire doc- ument collection, which are estimated as being most informative to learn an effective classifica- tion model (Cohn et al., 1996). This intentional selection bias stands in stark contrast to prevailing sampling approaches where annotation examples are randomly chosen. When different approaches to AL are compared with each other, or with standard random sam- pling, in terms of annotation efficiency, up until now, the AL community assumed uniform annota- tion …

Content truncated to 500 words