Information Retrieval Using Robust Natural Language Processing

INFORMATION RETRIEVAL USING ROBUST NATURAL LANGUAGE PROCESSING Tomek Strzalkowski and Barbara Vauthey1" Courant Institute of Mathematical Sciences New York University 715 Broadway, rm. 704 New York, NY 10003 tomek@cs.nyu.edu ABSTRACT We developed a prototype information retrieval sys- tem which uses advanced natural language process- ing techniques to enhance the effectiveness of tradi- tional key-word based document retrieval. The back- bone of our system is a statistical retrieval engine which performs automated indexing of documents, then search and ranking in response to user queries. This core architecture is augmented with advanced natural language processing tools which are both robust and efficient. In early experiments, the aug- mented system has displayed capabilities that appear to make it superior to the purely statistical base. INTRODUCTION A typical information retrieval fiR) task is to select documents from a database in response to a user's query, and rank these documents according to relevance. This has been usually accomplished using statistical methods (often coupled with manual encoding), but it is now widely believed that these traditional methods have reached their limits. 1 These limits are particularly acute for text databases, where natural language processing (NLP) has long been considered necessary for further progress. Unfor- tunately, the difficulties encountered in applying computational linguistics technologies to text pro- cessing have contributed to a wide-spread belief that automated NLP may not be suitable in IR. These difficulties included inefficiency, limited coverage, and prohibitive cost of manual effort required to build lexicons and knowledge bases for each new text domain. On the other hand, while numerous experiments did not establish the usefulness of NLP, they cannot be considered conclusive because of their very limited scale. Another reason is the limited scale at which NLP was used. Syntactic parsing of the database con- tents, for example, has been attempted in order to extract linguistically motivated "syntactic phrases", which presumably were better indicators of contents than "statistical phrases" where words were grouped solely on the basis of physical proximity (eg. "college junior" is not the same as "junior college"). These intuitions, however, were not confirmed by experi- ments; worse still, statistical phrases regularly out- performed syntactic phrases (Fagan, 1987). Attempts to overcome the poor statistical behavior of syntactic phrases has led to various clustering techniques that grouped synonymous or near synonymous phrases into "clusters" and replaced these by single "meta- terms". Clustering techniques were somewhat suc- cessful in upgrading overall system performance, but their effectiveness was diminished by frequently poor quality of syntactic analysis. Since full-analysis wide-coverage syntactic parsers were either unavail- able or inefficient, various partial parsing methods have been used. Partial parsing was usually fast enough, but it also generated noisy data_" as many as 50% of all generated phrases could be incorrect (Lewis and Croft, 1990). Other efforts concentrated on processing of user queries (eg. Spack Jones and Tait, 1984; Smeaton and van Rijsbergen, 1988). Since queries were usually short and few, even rela- tively inefficient NLP techniques could be of benefit to the system. None of these attempts …

Content truncated to 500 words