Categorization and Standardizing Proper Nouns for Efficient Information Retrieval

CATEGORIZING AND STANDARDIZING PROPER NOUNS FOR EFFICIENT INFORMATION RETRIEVAL Woojin Paik 1, Elizabeth D. Liddy 1, Edmund Yu 2, Mary McKenna 1 ! School of Inf(rmation Studies Syracuse University Syracuse, NY 13244 { wjpaik,fiddy,memckenn } @mailbex.syr.edu 2 College of Engineering and Computer Science Syracuse University Syracuse, NY 13244 esyu@mailbox.syr.edu Abstract In this paper, we describe the most recent implementation and evaluation of the proper noun categorization and standardization module of the DR- LINK document detection system being developed at Syracuse University, under the auspices of ARPA's TIPSTER program. We also discuss the expansion of group common nouns and group proper nouns to enhance retrieval recall. Successful proper noun boundary identification within the part of speech tagger is essential for successful categorization. The proper noun classification module is designed to assign a category code to each proper noun entity, using 30 categories generated from corpus analysis. Standardization of variant proper nouns occurs at three levels of processing. Expansion of group proper nouns and group common nouns is performed on queries. Standardization and categorization is performed on queries and documents. DR-LINK's overall precision for proper noun categorization was 93%, based on 589 proper nouns occurring in the evaluation set. 1. Introduction In information retrieval, proper nouns, group proper nouns, and group common nouns present unique problems. Proper nouns are recognized as an imp(xlant source of information for detecting relevant documents in information retrieval and extracting contents from a text (Ran, 1991). Yet most of the unknown words in texts which degrade the performance of natural language processing information retrieval systems are proper nouns. Group proper nouns (e.g., Middle East) and group common nouns (e.g., third world) will not match on their constituents unless the group entity is mentioned in the document. The proper noun processor herein described is a module in the DR-LINK system (Liddy et at, in press) for document detection being developed under the auspices of ARPA's TIPSTER program. Our approach to solving the group common noun and the group proper noun problem has been to expand the appropriate terms in a query, such as 'third world,' to all possible names and variants of third world entities. For all proper nouns, our system assigns categories from a proper noun classification scheme to every proper noun in both documents and queries to permit proper noun matching at the category level. Category matching is more efficient than keyword matching if the query requires entities of a particular type. Standardization provides a means of efficiently categorizing and retrieving documents containing variant forms of a proper noun. 2. Proper Noun Boundary Identification In our most recent implementation, which has improved from our initial attempt (Paik et at, in press), documents are first processed using a probabilistic part of speech tagger (Meeter et al, 1991). Then a proper noun boundary identifier utilizes proper noun part of speech tags from the previous stage to bracket adjacent proper nouns. Additionally, heuristics developed through corpus analysis are applied to bracket proper noun phrases with embedded conjunctions and prepositions …

Content truncated to 500 words