Building Effective Queries In Natural Language Information Retrieval

Building Effective Queries In Natural Language Information Retrieval Tomek Strzalkowski 1, Fang Lin 1, Jose Perez-Carballo 2 and Jin Wang 1 1GE Corporate Research & Development 1 Research Circle, Niskayuna, NY 12309 2School of Communication, Information and Library Studies, Rutgers University 4 Huntington Street, New Brunswick, NJ 08903 ABSTRACT In this paper we report on our natural language informa- tion retrieval (NLIR) project as related to the recently concluded 5th Text Retrieval Conference (TREC-5). The main thrust of this project is to use natural language processing techniques to enhance the effectiveness of full-text document retrieval. One of our goals was to demonstrate that robust if relatively shallow NLP can help to derive a better representation of text documents for statistical search. Recently, we have turned our attention away from text representation issues and more towards query development problems. While our NLIR system still performs extensive natural language pro- cessing in order to extract phrasal and other indexing terms, our focus has shifted to the problems of building effective search queries. Specifically, we are interested in query construction that uses words, sentences, and entire passages to expand initial topic specifications in an attempt to cover their various angles, aspects and contexts. Based on our earlier results indicating that NLP is more effective with long, descriptive queries, we allowed for long passages from related documents to be liberally imported into the queries. This method appears to have produced a dramatic improvement in the perfor- mance of two different statistical search engines that we tested (Cornell's SMART and NIST's Prise) boosting the average precision by at least 40%. In this paper we discuss both manual and automatic procedures for query expansion within a new stream-based information retrieval model. 1. INTRODUCTION A typical (full-text) information retrieval (IR) task is to select documents from a database in response to a user's query, and rank these documents according to relevance. This has been usually accomplished using statistical methods (often coupled with manual encoding) that (a) select terms (words, phrases, and other units) from doc- uments that are deemed to best represent their content, and (b) create an inverted index file (or files) that pro- vide an easy access to documents containing these terms. A subsequent search process will attempt to match preprocessed user queries against term-based rep- resentations of documents in each case determining a degree of relevance between the two which depends upon the number and types of matching terms. Although many sophisticated search and matching methods are available, the crucial problem remains to be that of an adequate representation of content for both the docu- ments and the queries. In term-based representation, a document (as well as a query) is transformed into a collection of weighted terms, derived directly from the document text or indi- rectly through thesauri or domain maps. The representa- tion is anchored on these terms, and thus their careful selection is critical. Since each unique term can be thought to add a new dimensionality to the representa- tion, it is …

Content truncated to 500 words