Word-Sense Distinguishability and Inter-Coder Agreement

Word-Sense Distinguishability and Inter-Coder Agreement * Rebecca Brucet and Janyce Wiebet t Department of Computer Science University of North Carolina at Asheville Asheville, NC 28804-8511 tDepartment of Computer Science New Mexico State University, Las Cruces, NM 88003 bruce@cs.unca.edu, wiebe@cs.nmsu.edu Abstract It. is common in NLP that the categories into which text is classified do not have fully objective def- initions. Examples of such categories are lexical distinctions such as part-of-speech tags and word- sense distinctions, sentence level distinctions such as phrase attachment, and discourse level distinc- t.icms such as topic or speech-act categorization. This p>1per presents an approach to analy?-ing the agrcen1ent arnong lnnnan judges for the purpose of formulating a refined and more reliable set of category designations. We use these techniques to analyze the sense tags assigned by five judgps to the noun intcr·est. The initial tag set is takmi from Longman's Dictionary of Contemporary }i:nglish. Through this process of analysis, we automatically identify and assign a revised set of sense tags for the data. The revised tags exhibit high reliabil- ity as measured by Cohen's r;.. Such techniques are important for formulating and evaluating both human and automated classification systems. Introduction It is common in Natural Language Processing (NLP) that the categories into which text is classi- fied do not have fully objective definitions. Exam- ples of such categories are lexical distinctions such as part-of-speech tags and word-sense distinctions, sentence level distinctions such as phrase attach- ment, and discourse level distinctions such as topic or speech-ac.t categorization. This paper presents an approach to analyzing the agreement among hu- man judges for the purpose of formulating a refined 1~his research was supported by the Of-fice of Naval Re- search under grant number N00014-95-l-0776. 53 and more reliable set of category designations. We performed a case study of the classification process, involving multiple judges performing a word-sense disambiguation task. Table 1 presents the data for two judges assigning one of six senses to each instance of inter-est used as a noun in the corpus. The data is represented as a contingency table, often referred to as a confusion matrix; it depicts the "confusion" among the judges' classi- fications. Evidence of eonfusion among the classi- fications in Table 1 can be found in the marginal totals, ni+ and n+.i> where i and j range h·om 1 to 6. We see that, on average, judge A has a higher preference for senses 1 and 3 than judge E does, while judge E has a higher preference for sense 2 than judge A does. These biases are one aspect of agreement (or the lack of it) among judges. A seeond aspect of agreement is the extent to which judges agree on the tags of individual words (mtegory distinguishability). We see from the diag- onal frequencies in Table 1 that these judges agree on 2097 out of 2369 of them, which is 88.5% of the individual tags. Cohen (1960) proposed the coefficient of agree- ment, r;, for measuring the agreement between two judges. …

Content truncated to 500 words