Robust Text Processing in Automated Information Retrieval

Robust Text Processing in Automated Information Retrieval Tomek Strzalkowski Courant Institute of Mathematical Sciences New York University 715 Broadway, rm. 704 New York, NY 10003 tomek@cs.nyu.edu Abstract We report on the results of a series of experiments with a prototype text retrieval system which uses relatively advanced natural language processing techniques in order to enhance the effectiveness of statistical docu- ment retrieval. In this paper we show that large-scale natural language processing (hundreds of millions of words and more) is not only required for a better retrieval, but it is also doable, given appropriate resources. In particular, we demonstrate that the use of syntactic compounds in the representation of database documents as well as in the user queries, coupled with an appropriate term weighting strategy, can consider- ably improve the effectiveness of retrospective search. The experiments reported here were conducted on TIP- STER database in connection with the Text REtrieval Conference series (TREC). 1 1 Introduction The task of information retrieval is to extract relevant documents from a large collection of docu- ments in response to user queries. When the docu- ments contain primarily unrestricted text (e.g., newspa- per articles, legal documents, etc.) the relevance of a document is established through 'full-text' retrieval. This has been usually accomplished by identifying key terms in the documents (the process known as 'index- ing') which could then be matched against terms in queries (Salton, 1989). The effectiveness of any such term-based approach is directly related to the accuracy with which a set of terms represents the content of a document, as well as how well it contrasts a given document with respect to other documents. In other words, we are looking for a representation R such that for any text items D1 and D2, R(D1) = R(D2) iff meaning(D1) = meaning(D2), at an appropriate level of abstraction (which may depend on the types and character of anticipated queries), ' See (Harman, 1993) for a detailed description of TREC. The simplest word-based representations of con- tent are usually inadequate since single words are rarely specific enough for accurate discrimination, and their grouping is often accidental. A better method is to identify groups of words that create meaningful phrases, especially if these phrases denote important concepts in the database domain. For example, joint venture is an important term in the Wall Street Journal (WSJ henceforth) database, while neither joint nor ven- ture are important by themselves. In fact, in a 800+ MBytes database, both joint and venture would often be dropped from the list of terms by the system because their inverted document frequency (idj) weights were too low. In large databases comprising hundreds of thousands of documents the use of phrasal terms is not just desirable, it becomes necessary. To illustrate this point let us consider TREC Topic 104, an information request from which a data- base search query is to be built. The reader may note various sections of this Topic, with <desc> correspond- ing to the user's original request, further elaborated in …

Content truncated to 500 words