Fast Statistical Parsing of Noun Phrases for Document Indexing

Fast Statistical Parsing of Noun Phrases for Document Indexing Chengxiang Zhai Laboratory for Computational Linguistics Carnegie Mellon University Pittsburgh, PA 15213 {cz25@andrew.cmu.edu} Abstract Information Retrieval (IR) is an important application area of Natural Language Pro- cessing (NLP) where one encounters the genuine challenge of processing large quan- tities of unrestricted natural language text. While much effort has been made to apply NLP techniques to IR, very few NLP tech- niques have been evaluated on a document collection larger than several megabytes. Many NLP techniques are simply not ef- ficient enough, and not robust enough, to handle a large amount of text. This pa- per proposes a new probabilistic model for noun phrase parsing, and reports on the application of such a parsing technique to enhance document indexing. The effective- ness of using syntactic phrases provided by the parser to supplement single words for indexing is evaluated with a 250 megabytes document collection. The experiment's re- sults show that supplementing single words with syntactic phrases for indexing consis- tently and significantly improves retrieval performance. 1 Introduction Information Retrieval (IR) is an increasingly impor- tant application area of Natural Language Process- ing (NLP). An IR task can be described as to find, from a given document collection, a subset of docu- ments whose content is relevant to the information need of a user as expressed by a query. As the doc- uments and query are often natural language texts, an IR task can usually be regarded as a special NLP task, where the document text and the query text need to be processed in order to judge the relevancy. A general strategy followed by most IR systems is to transform documents and the query into certain level of representation. A query representation can then be compared with a document representation to decide if the document is relevant to the query. In practice, the level of representation in an IR system is quite "shallow" -- often merely a set of word-like strings, or indexing terms. The process to extract in- dexing terms from each document in the collection is called indexing. A query is often subject to simi- lax processing, and the relevancy is judged based on the matching of query terms and document terms. In most systems, weights are assigned to terms to indicate how well they can be used to discriminate relevant documents from irrelevant ones. The challenge in applying NLP to IR is to deal with a large amount of unrestricted natural lan- guage text. The NLP techniques used must be very efficient and robust, since the amount of text in the databases accessed is typically measured in gi- gabytes. In the past, NLP techniques of different levels, including morphological, syntactic/semantic, and discourse processing, were exploited to enhance retrieval (Smeaton 92; Lewis and Spaxck Jones 96), but were rarely evaluated using collections of docu- ments larger than several megabytes. Many NLP techniques are simply not efficient enough or are too labor-intensive to successfully handle a large size document set. However, …

Content truncated to 500 words