Investigating the Effects of Selective Sampling on the Annotation Task

Proceedings of the 9th Conference on Computational Natural Language Learning (CoNLL), pages 144–151, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Investigating the Effects of Selective Sampling on the Annotation Task Ben Hachey, Beatrice Alex and Markus Becker School of Informatics University of Edinburgh Edinburgh, EH8 9LW, UK {bhachey,v1balex,s0235256}@inf.ed.ac.uk Abstract We report on an active learning experi- ment for named entity recognition in the astronomy domain. Active learning has been shown to reduce the amount of la- belled data required to train a supervised learner by selectively sampling more in- formative data points for human annota- tion. We inspect double annotation data from the same domain and quantify poten- tial problems concerning annotators’ per- formance. For data selectively sampled according to different selection metrics, we find lower inter-annotator agreement and higher per token annotation times. However, overall results confirm the util- ity of active learning. 1 Introduction Supervised training of named entity recognition (NER) systems requires large amounts of manually annotated data. However, human annotation is typ- ically costly and time-consuming. Active learn- ing promises to reduce this cost by requesting only those data points for human annotation which are highly informative. Example informativity can be estimated by the degree of uncertainty of a single learner as to the correct label of a data point (Cohn et al., 1995) or in terms of the disagreement of a committee of learners (Seung et al., 1992). Ac- tive learning has been successfully applied to a va- riety of tasks such as document classification (Mc- Callum and Nigam, 1998), part-of-speech tagging (Argamon-Engelson and Dagan, 1999), and parsing (Thompson et al., 1999). We employ a committee-based method where the degree of deviation of different classifiers with re- spect to their analysis can tell us if an example is potentially useful. In a companion paper (Becker et al., 2005), we present active learning experiments for NER in radio-astronomical texts following this approach.1 These experiments prove the utility of selective sampling and suggest that parameters for a new domain can be optimised in another domain for which annotated data is already available. However there are some provisos for active learn- ing. An important point to consider is what effect informative examples have on the annotators. Are these examples more difficult? Will they affect the annotators’ performance in terms of accuracy? Will they affect the annotators performance in terms of time? In this paper, we explore these questions us- ing doubly annotated data. We find that selective sampling does have an adverse effect on annotator accuracy and efficiency. In section 2, we present standard active learn- ing results showing that good performance can be achieved using fewer examples than random sam- pling. Then, in section 3, we address the questions above, looking at the relationship between inter- annotator agreement and annotation time and the ex- amples that are selected by active learning. Finally, section 4 presents conclusions and future work. 1Please refer to the companion paper for details of the selective sampling approach with experimental adaptation …

Content truncated to 500 words