Noun Phrase Analysis in Large Unrestricted Text for Information Retrieval

Noun-Phrase Analysis in Unrestricted Text for Information Retrieval David A. Evans, Chengxiang Zhai Laboratory for Computational Linguistics Carnegie Mellon Univeristy Pittsburgh, PA 15213 dae@cmu.edu, cz25@andrew.cmu.edu Abstract Information retrieval is an important ap- plication area of natural-language pro- cessing where one encounters the gen- uine challenge of processing large quanti- ties of unrestricted natural-language text. This paper reports on the application of a few simple, yet robust and efficient noun- phrase analysis techniques to create bet- ter indexing phrases for information re- trieval. In particular, we describe a hy- brid approach to the extraction of mean- ingful (continuous or discontinuous) sub- compounds from complex noun phrases using both corpus statistics and linguistic heuristics. Results of experiments show that indexing based on such extracted sub- compounds improves both recall and pre- cision in an information retrieval system. The noun-phrase analysis techniques are also potentially useful for book indexing and automatic thesaurus extraction. 1 Introduction 1.1 Information Retrieval Information retrieval (IR) is an important applica- tion area of naturaManguage processing (NLP). 1 The IR (or perhaps more accurately "text retrieval") task may be characterized as the problem of select- ing a subset of documents (from a document col- lection) whose content is relevant to the informa- tion need of a user as expressed by a query. The document collections involved in IR are often gi- gabytes of unrestricted natural-language text. A user's query may be expressed in a controlled lan- guage (e.g., a boolean expression of keywords) or, more desirably, a natural language, such as English. A typical IR system works as follows. The doc- uments to be retrieved are processed to extract in- dexing terms or content carriers, which are usually (Evans, 1990; Evans et al., 1993; Smeaton, 1992; Lewis & Sparck Jones, 1996) single words or (less typically) phrases. The index- ing terms provide a description of the document's content. Weights are often assigned to terms to in- dicate how well they describe the document. A (natural-language) query is processed in a similar way to extract query terms. Query terms are then matched against the indexing terms of a document to determine the relevance of each document to the quer3a The ultimate goal of an IR system is to increase both precision, the proportion of retrieved docu- ments that are relevant, as well as recall, the propor- tion of relevant document that are retrieved. How- ever, the real challenge is to understand and rep- resent appropriately the content of a document and quer~ so that the relevance decision can be made ef- ficiently, without degrading precision and recall. A typical solution to the problem of making relevance decisions efficient is to require exact matching of in- dexing terms and query terms, with an evaluation of the 'hits' based on a scoring metric. Thus, for instance, in vector-space models of relevance rank- ing, both the indexing terms of a document and the query terms are treated as vectors (with individual term weights) and the similarity between the two vectors is given by …

Content truncated to 500 words