Linguistic Knowledge can Improve Information Retrieval

Linguistic Knowledge can Improve Information Retrieval William A. Woods and Lawrence A. Bookman* and Ann Houston and Robert J. Kuhns and Paul Martin and Stephen Green Sun Microsystems Laboratories 1 Network Drive Burlington, MA 01803 {William.Woods,Ann.Houston,Robert.Kuhns,Paul.Martin,Stephen.Green}@east.sun.com Abstract This paper describes the results of some experiments using a new approach to information access that combines techniques from natural language process- ing and knowledge representation with a penalty- based technique for relevance estimation and passage retrieval. Unlike many attempts to combine natural language processing with information retrieval, these results show substantial benefit from using linguistic knowledge. 1 Introduction An online information seeker often fails to find what is wanted because the words used in the request are different from the words used in the relevant mate- rial. Moreover, the searcher usually spends a signifi- cant amount of time reading retrieved material in or- der to determine whether it contains the information sought. To address these problems, a system has been developed at Sun Microsystems Laboratories (Ambroziak and Woods, 1998) that uses techniques from natural language processing and knowledge representation, with a technique for dynamic pas- sage selection and scoring, to significantly improve retrieval performance. This system is able to locate specific passages in the indexed material where the requested information appears to be, and to score those passages with a penalty-based score that is highly correlated with the likelihood that they con- tain relevant information. This ability, which we call "Precision Content Retrieval" is achieved by com- bining a system for Conceptual Indexing with an algorithm for Relaxation-Ranking Specific Passage Retrieval. In this paper, we show how linguistic knowledge is used to improve search effectiveness in this system. This is of particular interest, since many previous at- tempts to use linguistic knowledge to improve infor- mation retrieval have met with little or mixed suc- cess (Fagan, 1989; Lewis and Sparck Jones, 1996; Sparck Jones, 1998; Varile and Zampolli, 1997; Voorhees, 1993; Mandala et al., 1999) (but see the latter for some successes as well). * Lawrence Bookman is now at Torrent Systems, Inc. 2 Conceptual Indexing The conceptual indexing and retrieval system used for these experiments automatically extracts words and phrases from unrestricted text and organizes them into a semantic network that integrates syn- tactic, semantic, and morphological relationships. The resulting conceptual taxonomy (Woods, 1997) is used by a specific passage-retrieval algorithm to deal with many paraphrase relationships and to find spe- cific passages of text where the information sought is likely to occur. It uses a lexicon containing syntac- tic, semantic, and morphological information about words, word senses, and phrases to provide a base source of semantic and morphological relationships that are used to organize the taxonomy. In addi- tion, it uses an extensive system of knowledge-based morphological rules and functions to analyze words that are not already in its lexicon, in order to con- struct new lexical entries for previously unknown words (Woods, 2000). In addition to rules for han- dling derived and inflected forms of known words, the system …

Content truncated to 500 words