The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76–83, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account Massimo Poesio and Ron Artstein University of Essex, Language and Computation Group / Department of Computer Science United Kingdom Abstract We report the results of a study of the reliability of anaphoric annotation which (i) involved a substantial number of naive subjects, (ii) used Krippendorff’s α in- stead of K to measure agreement, as re- cently proposed by Passonneau, and (iii) allowed annotators to mark anaphoric ex- pressions as ambiguous. 1 INTRODUCTION We tackle three limitations with the current state of the art in the annotation of anaphoric relations. The first problem is the lack of a truly systematic study of agreement on anaphoric annotation in the literature: none of the studies we are aware of (Hirschman, 1998; Poesio and Vieira, 1998; Byron, 2003; Poe- sio, 2004) is completely satisfactory, either because only a small number of coders was involved, or because agreement beyond chance couldn’t be as- sessed for lack of an appropriate statistic, a situation recently corrected by Passonneau (2004). The sec- ond limitation, which is particularly serious when working on dialogue, is our still limited understand- ing of the degree of agreement on references to ab- stract objects, as in discourse deixis (Webber, 1991; Eckert and Strube, 2001). The third shortcoming is a problem that affects all types of semantic annotation. In all annotation stud- ies we are aware of,1 the fact that an expression may not have a unique interpretation in the context of its 1The one exception is Rosenberg and Binkowski (2004). occurrence is viewed as a problem with the anno- tation scheme, to be fixed by, e.g., developing suit- ably underspecified representations, as done partic- ularly in work on wordsense annotation (Buitelaar, 1998; Palmer et al., 2005), but also on dialogue act tagging. Unfortunately, the underspecification solu- tion only genuinely applies to cases of polysemy, not homonymy (Poesio, 1996), and anaphoric ambigu- ity is not a case of polysemy. Consider the dialogue excerpt in (1):2 it’s not clear to us (nor was to our annotators, as we’ll see below) whether the demon- strative that in utterance unit 18.1 refers to the ‘bad wheel’ or ‘the boxcar’; as a result, annotators’ judg- ments may disagree – but this doesn’t mean that the annotation scheme is faulty; only that what is being said is genuinely ambiguous. (1) 18.1 S: .... 18.6 it turns out that the boxcar at Elmira 18.7 has a bad wheel 18.8 and they’re .. gonna start fixing that at midnight 18.9 but it won’t be ready until 8 19.1 M: oh what a pain in the butt This problem is encountered with all types of anno- tation; the view that all types of disagreement indi- cate a problem with the annotation scheme–i.e., that somehow the problem would disappear if only we could find the right annotation scheme, …

Content truncated to 500 words