Inconsistency Detection in Semantic Annotation

Inconsistency Detection in Semantic Annotation Nora Hollenstein, Nathan Schneider, Bonnie Webber University of Edinburgh School of Informatics s1303835@ed.ac.uk, nschneid@inf.ed.ac.uk, bonnie@inf.ed.ac.uk Abstract Inconsistencies are part of any manually annotated corpus. Automatically finding these inconsistencies and correcting them (even manually) can increase the quality of the data. Past research has focused mainly on detecting inconsistency in syntactic annotation. This work explores new approaches to detecting inconsistency in semantic annotation. Two ranking methods are presented in this paper: a discrepancy ranking and an entropy ranking. Those methods are then tested and evaluated on multiple corpora annotated with multiword expressions and supersense labels. The results show considerable improvements in detecting inconsistency candidates over a random baseline. Possible applications of methods for inconsistency detection are improving the annotation procedure and guidelines, as well as correcting errors in completed annotations. Keywords: inconsistency detection, annotation, error detection, corpora 1. Introduction Inconsistencies can be found in any manually-annotated corpus due to underspecified or even missing guidelines, ambiguity, insufficient annotator expertise, and/or human errors. A consistent corpus is not only a high-quality lexical resource, but a foundation for robust automatic language processing via supervised learning. Since inconsistencies in the training corpus can lead to low performance, consistent annotation is of practical as well as theoretical benefit. Annotation inconsistencies can be subclassified into annotation errors and hard cases, although they cannot always be easily distinguished (Plank et al., 2014). An annotation error is an instance that is annotated incor- rectly according to the guidelines. Annotation errors have a correct answer. To give an example from part-of-speech annotation, consider the phrase ‘a summer feeling’. The word ‘summer’ should be annotated as a common noun; marking it as a comparative adjective or a determiner would be an annotation error. Linguistically hard cases are instances that do not have one correct answer; they may have multiple correct answers or it may not be obvious what the label should be. Such annotations are not necessarily incorrect, but can be ambiguous or underspecified in the guidelines. Beigman Klebanov and Beigman (2014) define hard cases as instances which are difficult to decide upon, which is when annotator preferences come into play, and as a consequence inter-annotator agreement and the consistency in the corpus are lower than in linguistically unambiguous cases. Our concern here is the automatic detection of inconsis- tencies in semantic annotation. We consider datasets anno- tated for lexical semantic units (multiword expressions) and classes (supersenses); Section 2 gives an overview of these annotations. To rank lexical types that may be annotated inconsistently, we propose two methods, one based on ab- solute frequency and a second based on entropy (Section 3). Section 4 presents a precision-oriented evaluation of these methods on the lexical semantic datasets; Section 5 argues that these results indicate the viability of our methods for helping improve corpus consistency. We conclude with a discussion of related and future work. 2. Data sets 2.1. Multiword Expressions Multiword expressions (MWEs) are expressions of at least two words whose combination is syntactically and/or semantically idiosyncratic …

Content truncated to 500 words