Evaluation Techniques for Automatic Semantic Extraction: Comparing Syntactic and Window Based Approaches

Evaluation Techniques for Automatic Semantic Extraction: Comparing Syntactic and Window Based Approaches Gregory Grefenstette Department of Computer Science University of Pittsburgh Pittsburgh, PA 15260 grefen@cs.pitt, edn Abstract As large on-line corpora become more prevalent, a number of attempts have been made to automatically extract thesaurus-like relations directly from text using knowl- edge poor methods. In the absence of any specific application, comparing the results of these attempts is difficult. Here we propose an evaluation method using gold stan- dards, i.e., pre-existing hand-compiled resources, as a means of comparing extraction techniques. Using this evaluation method, we compare two semantic extraction tech- niques which produce similar word lists, one using syntactic context of words , and the other using windows of heuristically tagged words. The two techniques are very similar except that in one case selective natural language processing, a partial syn- tactic analysis, is performed. On a 4 megabyte corpus, syntactic contexts produce significantly better results against the gold standards for the most characteristk: words in the corpus, while windows produce better results for rare words. 1 Introduction As more text becomes available electronically, it is tempting to imagine the development of automatic filters able to screen these tremendous flows of text extracting usefill bits of information. In order to properly filter, it is useful to know when two words are similar in a corpus. Knowing this would allcviate part of the term variability problem of natural language discussed in Furnas et al. (1987) . Individuals will choose a variety of words to name the same object or operation, with little overlap between people's choices. This variability in naming was cited as the principal reason for large numbers of missed citations in a large-scale evaluation of an information retrieval system [Blair and Maron, 1985]. A proper filter must be able to access information in the text using any word of a set of similar words. A number of knowledge-rich [Jacobs and Rau, 1990, Calzolari and Bindi, 1990, Mauldin, 1991] and knowledge-poor [Brown et al., 1992, Hindle, 1990, Ruge, 1991, Grefenstette, 1992] methods have been proposed for recognizing when words are similar. The knowledge-rich approaches require either a conceptual dependency representation, or semantic tagging of the words, while the knowledge-poor approaches require no previously encoded semantic information, and depend on frequency of co-occurrence of word contexts to determine similarity. Evaluations of results produced by the above systems are often been limited to visual verification by a human subject or left to the human reader. In this paper, we propose gold standard evaluation techniques, allowing us to ob- jectively evaluate and to compare two knowledge-poor approaches for extracting word similarity relations from large text corpora. In order to evaluate the relations extracted, we measure the overlap of the results of each technique against existing hand-created 143 repositories of semantic information such as thesauri and dictionaries. We describe below }low such resources can be used as evaluation tools, and apply them to two knowledge-poor approaches. One of the tested semantic extraction approaches uses selective …

Content truncated to 500 words