The Benefits of a Model of Annotation

The Benefits of a Model of Annotation Rebecca J. Passonneau Center for Computational Learning Systems Columbia University New York, NY USA becky@ccls.columbia.edu Bob Carpenter Department of Statistics Columbia University New York, NY USA carp@alias-i.com Abstract Standard agreement measures for interannota- tor reliability are neither necessary nor suffi- cient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained an- notators are contrasted with a probabilistic an- notation model applied to crowdsourced data. The annotation model provides far more in- formation, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach. 1 Introduction The quality of annotated data for computational lin- guistics is generally assumed to be good enough if a few annotators can be shown to be consistent with one another. Standard practice relies on met- rics that measure consistency, either in an absolute way, or in a chance-adjusted fashion. Such mea- sures, however, merely report how often annota- tors agree, with no direct measure of corpus qual- ity, nor of the quality of individual items. We ar- gue that high chance-adjusted interannotator agree- ment is neither necessary nor sufficient to ensure high quality gold-standard labels. We contrast the use of agreement metrics with the use of probabilis- tic models to draw inferences about annotated data where the items have been labeled by many anno- tators. A probabilistic model to fit many annota- tors’ observed labels produces much more informa- tion about the annotated corpus. In particular, there will be a confidence estimate for each ground truth label. Probabilistic models of agreement and gold- standard inference have been used in psychomet- rics and marketing since the 1950s (e.g., IRT mod- els or Bradley-Terry models) and in epidemiology since the 1970s (e.g., diagnostic disease prevalence models). More recently, crowdsourcing has moti- vated their application to data annotation for ma- chine learning. The model we apply here (Dawid and Skene, 1979) assumes that annotators differ from one another in their accuracy at identifying the true label values, and that these true values occur at certain rates (their prevalence). To contrast the two approaches to creation of an annotated corpus, we present a case study of word sense annotation. The items that were annotated are occurrences of words in their sentence contexts, and each label is a WordNet sense (Miller, 1995). Each item has sense labels from up to twenty-five different annotators, collected through crowdsourcing. Ap- plication of an annotation model does not require this many labels per item, and crowdsourced annota- tion data does not require a probabilistic model. The case study, however, shows how the two benefit each other. MASC (Manually Annotated Sub-Corpus of the Open American National Corpus) contains a sub- sidiary word sense sentence corpus that consists of approximately one thousand sentences per word for 116 words. Word senses were annotated in their sen- tence contexts using WordNet sense labels. Chance- adjusted agreement levels …

Content truncated to 500 words