A Layered Approach to NLP-Based Information Retrieval

A LAYERED APPROACH TO NLP-BASED INFORMATION RETRIEVAL Sharon Flank SRA International 4300 Fair Lakes Court Fairfax, VA 22033, USA flanks~sra.com Abstract A layered approach to information retrieval permits the inclusion of multiple search en- gines as well as multiple databases, with a natural language layer to convert English queries for use by the various search en- gines. The NLP layer incorporates mor- phological analysis, noun phrase syntax, and semantic expansion based on Word- Net. 1 Introduction This paper describes a layered approach to infor- mation retrieval, and the natural language compo- nent that is a major element in that approach. The layered approach, packaged as Intermezzo TM, was deployed in a pre-product form at a government site. The NLP component has been installed, with a proprietary IR engine, PhotoFile, (Flank, Martin, Balogh and Rothey, 1995), (Flank, Garfield, and Norkin, 1995), at several commercial sites, includ- ing Picture Network International (PNI), Simon and Schuster, and John Deere. Intermezzo employs an abstraction layer to per- mit simultaneous querying of multiple databases. A user enters a query into a client, and the query is then passed to the server. The abstraction layer, part of the server, converts the query to the ap- propriate format for each of the databases (e.g. Fulcrum TM, RetrievalWare TM, Topic TM, WAIS). In Boolean mode, queries are translated, using an SGML-based intermediate query language, into the appropriate form; in NLP mode the queries un- dergo morphological analysis, NP syntax, and se- mantic expansion before being converted for use by the databases. The following example illustrates how a user's query is translated. Unexpanded query natural disasters in New England Search-engine specific natural AND disaster(s) AND New AND England Semantic expansion ((natural and disaster(s)) or hurricane(s) or earthquake(s) or tornado(es) in ("New England" or Maine or Vermont or "New Hampshire" or "Rhode Island" or Connecticut or Massachusetts) The NLP component has been deployed with as many as 500,000 images, at Picture Network In- ternational (PNI). The original commercial use of PNI was as a dialup system, launched with ap- proximately 100,000 images. PNI now operates on the World Wide Web (www.publishersdepot.com). Adjustment of the NLP component continued ac- tively up through about 250,000 images, including additions to the semantic net and tuning of the parameters for weighting. Retrieval speed for the NLP component averages under a second. Semantic expansion is performed in advance on the caption database, not at runtime; runtime expansion makes operation too slow. The remainder of this paper describes how the NLP mode works, and what was required to create it. 2 The NLP Techniques The natural language processing techniques used in this system are well known, including in infor- mation retrieval applications (Strzalkowski, 1993), (Strzalkowski, Perez Carballo and Marinescu, 1995), (Evans and Zhai, 1996). The importance of this work lies in the scale and robustness of the tech- niques as combined into a system for querying large databases. The NLP component is also layered, in effect. It uses a conventional search algorithm (several were tested, and …

Content truncated to 500 words