Customizing a Lexicon to Better Suit a Computational Task

Customizing a Lexicon to Better Suit a Computational Task Marti A. Hearst Hinrich Schfitze Computer Science Division 571 Evans Hall, UC Berkeley Berkeley, CA 94720 USA ma rti@cs, be rkele y. edu CSLI Ventura Hall, Stanford University Stanford, CA 94305 USA schuetze @csli. stanford, edu Abstract We discuss a method for augmenting and rearranging a structured lexicon in order to make it more suitable for a topic labefing task, by making use of lexical association information from a large text corpus. We first describe an algorithm for converting the hierarchical structure of WordNet [13] into a set of flat categories. We then use lexical cooccurrence statistics in combination with these categories to classify proper names, assign more specific senses to broadly defined terms, and classify new words into existing categories. We also describe how to use these statistics to assign schema-like information to the categories and show how the new categories improve a text-labeling algorithm. In effect, we provide a mechanism for successfully combining a hand-built lexicon with knowledge-free, statistically-derived information. 1 Introduction Much effort is being applied to the creation of lexicons and the acquisition of semantic and syntactic attributes of the lexical items that comprise them, e.g, [1], [4],[7],[8], [11], [16], [18], [20]. However, a lexicon as given may not suit the requirements of a particular computational task. Because lexicons are expensive to build, rather than create new ones from scratch, it is preferable to adjust existing ones to meet an application's needs. In this paper we describe such an effort: we add associational information to a hierarchically structured lexicon in order to better serve a text labeling task. An algorithm for partitioning a full-length expository text into a sequence of subtopieal discussions is described in [9]. Once the partitioning is done, we need to assign labels 1 indicating what the subtopical discussions are about, for the purposes of information retrieval and hypertext navigation. One way to label texts, when working within a limited domain of discourse, is to start with a pre-defined set of topics and specify the word contexts that indicate the topics of interest (e.g., [10]). Another way, assuming that a large collection of pre-labeled texts exists, is to use statistics to automatically infer which lexical items indicate which labels (e.g., [12]). In contrast, we are interested in assigning labels to general, domain-independent text, without benefit of pre-classified texts. In all three cases, a lexicon that specifies which lexical items correspond to which topics is required. The topic labeling method we use is statistical and thus requires a large number of representative lexical items for each category. The starting point for our lexicon is WordNet [13], which is readily available online and provides a large repository of English lexical items. WordNet 2 is composed of synsets, 1 The terms "label" and "topic" are used interchangeably in this paper. 2 All work described here pertains to Version 1.3 of WordNet. 55 structures containing sets of terms with synonymous meanings, thus allowing a distinction to be made …

Content truncated to 500 words