Evaluation of Semantic Clusters

Evaluation of Semantic Clusters Rajeev Agarwal Mississippi State University Mississippi State, MS 39762 USA rajeev@cs.msstate.edu Abstract Semantic clusters of a domain form an important feature that can be useful for performing syntactic and semantic disam- biguation. Several attempts have been made to extract the semantic clusters of a domain by probabilistic or taxonomic tech- niques. However, not much progress has been made in evaluating the obtained se- mantic clusters. This paper focuses on an evaluation mechanism that can be used to evaluate semantic clusters produced by a system against those provided by human experts. 1 Introduction 1 Most natural language processing (NLP) systems are designed to work on certain specific domains and porting them to other domains is often a very time- consuming and human-intenslve process. As the need for applying NLP systems to more and var- ied domains grows, it becomes increasingly impor- tant that some techniques be used to make these systems more portable. Several researchers (Lang and Hirschman, 1988; Rau et al., 1989; Pustejovsky, 1992; Grishman and Sterling, 1993; Basili et al., 1994), either directly or indirectly, have addressed issues that assist in making it easier to move an NLP system from one domain to another. One of the reasons for the lack of portability is the need for domain-specific semantic features that such systems often use for lexical, syntactic, and semantic disam- biguation. One such feature is the knowledge of the semantic clusters in a domain. Since semantic classes are often domain-specific, their automatic acquisition is not trivial. Such classes can be derived either by distributional means or from existing taxonomies, knowledge bases, dic- tionaries, thesauruses, and so on. A prime exam- ple of the latter is WordNet which has been used to 1The author is currently at Texas Instruments and all inquiries should be addressed to rajeev@csc.ti.com. provide such semantic classes (Resnik, 1993; Basili et al., 1994) to assist in text understanding. Our efforts to obtain such semantic clusters with limited human intervention have been described elsewhere (Agarwal, 1995). This paper concentrates on the aspect of evahiating the obtained clusters against classes provided by human experts. 2 The Need Although there has been a lot of work done in ex- tracting semantic classes of a given domain, rela- tively little attention has been paid to the task of evaluating the generated classes. In the absence of an evaluation scheme, the only way to decide if the semantic classes produced by a system are "reason- able" or not is by having an expert analyze them by inspection. Such informal evaluations make it very difficult to compare one set of classes against an- other and are also not very reliable estimates of the quality of a set of classes. It is clear that a formal evaluation scheme would be of great help. Hatzivassiloglou and McKeown (1993) duster ad- jectives into partitions and present an interest- ing evaluation to compare the generated adjective classes against those provided by an expert. Their evaluation scheme bases the comparison between two …

Content truncated to 500 words