Managing Uncertainty in Semantic Tagging

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 840–850, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Managing Uncertainty in Semantic Tagging Silvie Cinkov´a and Martin Holub and Vincent Kr´ıˇz Charles University in Prague, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {cinkova|holub}@ufal.mff.cuni.cz vincent.kriz@gmail.com Abstract Low interannotator agreement (IAA) is a well-known issue in manual semantic tag- ging (sense tagging). IAA correlates with the granularity of word senses and they both correlate with the amount of informa- tion they give as well as with its reliability. We compare different approaches to seman- tic tagging in WordNet, FrameNet, Prop- Bank and OntoNotes with a small tagged data sample based on the Corpus Pattern Analysis to present the reliable information gain (RG), a measure used to optimize the semantic granularity of a sense inventory with respect to its reliability indicated by the IAA in the given data set. RG can also be used as feedback for lexicographers, and as a supporting component of automatic se- mantic classifiers, especially when dealing with a very fine-grained set of semantic cat- egories. 1 Introduction The term semantic tagging is used in two diver- gent areas: 1) recognizing objects of semantic importance, such as entities, events and polarity, often tailored to a restricted domain, or 2) relating occurrences of words in a corpus to a lexicon and selecting the most appropriate seman- tic categories (such as synsets, semantic frames, wordsenses, semantic patterns or framesets). We are concerned with the second case, which seeks to make lexical semantics tractable for com- puters. Lexical semantics, as opposed to proposi- tional semantics, focuses the meaning of lexical items. The disciplines that focus lexical seman- tics are lexicology and lexicography rather than logic. By semantic tagging we mean a process of assigning semantic categories to target words in given contexts. This process can be either manual or automatic. Traditionally, semantic tagging relies on the tacit assumption that various uses of polysemous words can be sorted into discrete senses; under- standing or using an unfamiliar word be then like looking it up in a dictionary. When building a dic- tionary entry for a given word, the lexicographer sorts a number of its occurrences into discrete senses present (or emerging) in his/her mental lex- icon, which is supposed to be shared by all speak- ers of the same language. The assumed common mental representation of a words meaning should make it easy for other humans to assign random occurrences of the word to one of the pre-defined senses (Fellbaum et al., 1997). This assumption seems to be falsified by the interannotator agreement (IAA, sometimes ITA) constantly reported much lower in semantic than in morphological or syntactic annotation, as well as by the general divergence of opinion on which value of which IAA measure indicates a reliable annotation. In some projects (e.g. OntoNotes (Hovy et al., 2006)), the percentage of agreements between two annotators is used, but a …

Content truncated to 500 words