The Annotation Conundrum

Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics, page 2, Athens, Greece, 30 March, 2009. c⃝2009 Association for Computational Linguistics The Annotation Conundrum Mark Liberman University of Pennsylvania myl@cis.upenn.edu Abstract Without lengthy, iterative refinement of guidelines, and equally lengthy and itera- tive training of annotators, the level of in- ter-subjective agreement on simple tasks of phonetic, phonological, syntactic, se- mantic, and pragmatic annotation is shockingly low. This is a significant practical problem in speech and language technology, but it poses questions of interest to psycholo- gists, philosophers of language, and theo- retical linguists as well. 1 Introduction Biologists believe that they know what genes, organisms, chemical compounds, and diseases are. Linguists believe that they know what nouns, verbs, and claus- es are. Ordinary literate speakers of En- glish believe that they know what people, places, and organizations are. And all of them believe that they can recognize and understand instances of these categories in coherent text. When two biologists, two linguists, or two English speakers discuss such texts, it seems plausible that they have under- stood such instances in the same way. Nevertheless, if they are asked to high- light these instances, the level of inter- subjective agreement will be shockingly low. Similarly depressing results are obtained in tasks such as phonetic or surface- phonemic transcription, co-reference an- notation, identification of animacy, etc. Things are usually not much better if we compare annotations produced by the same individuals on different occasions. A solution exists, in the practical sense of producing annotations with high inter-an- notator agreement scores. The initially- divergent results of multiple annotations are discussed and adjudicated, and princi- ples of interpretation are defined for fu- ture use. This process is repeated over and over again, typically for several months, until the desired level of agree- ment is obtained, or funding runs out. At least for simple linguistic annotation tasks, this process, reminiscent of the de- velopment of common law, generally converges (though the residual level of disagreement may be depressingly high, especially when multiple judgments must be cascaded). The resulting annotation manuals may be hundreds of pages long, even for fairly limited tasks; and new an- notators face weeks or months of training to become competent in learning to apply them. There are several obvious ideas about why this might be true, but most of these ideas seem to be false. It will be argued that part of the answer lies in understand- ing that most linguistic annotation tasks are not really classification problems, but rather translation problems. We don’t normally assume that there is only one correct translation of a Chinese sentence into English; nor do we try to make this true by constructing elaborate translation guidelines to cover every relevant contin- gency, though in principle we could. Implications in engineering and science will be discussed. 2