Document Representation in Natural Language Text Retrieval

DOCUMENT REPRESENTATION IN NATURAL LANGUAGE TEXT RETRIEVAL Tomek Strzalkowski Courant Institute of Mathematical Sciences ABSTRACT In information retrieval, the content of a document may be represented as a collection of terms: words, stems, phrases, or other units derived or inferred from the text of the document. These terms are usually weighted to indicate their importance within the document which can then be viewed as a vector in a N- dimensional space. In this paper we demonstrate that a proper term weighting is at least as important as their selection, and that dif- ferent types of terms (e.g., words, phrases, names), and terms derived by different means (e.g., statistical, linguistic) must be treated differently for a maximum benefit in rel~ieval. We report some observations made during and after the second Text REtrieval Conference (TREC-2). 1 1. INTRODUCTION The task of information retrieval is to extract relevant documents from a large collection of documents in response to user queries. When the documents contain primarily unrestricted text (e.g., newspaper articles, legal documents, etc.) the relevance of a docu- ment is established through 'full-text' retrieval. This has been usu- ally accomplished by identifying key terms in the documents (the process known as 'indexing') which could then be matched against terms in queries [2]. The effectiveness of any such term-based approach is directly related to the accuracy with which a set of terms represents the content of a document, as well as how well it contrasts a given document with respect to other documents. In other words, we are looking for a representation R such that for any text items D1 and D2, R(D1) = R(D2) iff meaning(D1) = meaning(D2), at an appropriate level of abstraction (which may depend on the types and character of anticipated queries). The simplest word-based representations of content are usually inadequate since single words are rarely specific enough for accu- rate discrimination, and their grouping is often accidental. A better method is to identify groups of words that create meaningful phrases, especially if these phrases denote important concepts in the database domain. For example, joint venture is an important term in the Wall Street Journal (WSJ henceforth) database, while neither joint nor venture are important by themselves. In fact, in a 800+ MBytes database, both joint and venture would often be dropped from the list of terms by the system because their inverted document frequency (idJ) weights were too low. In large databases l See [1] for a detailed introduction to TREC. New York University 715 Broadway, rm. 704 New York, NY 10003 tomek@cs.nyu.edu comprising hundreds of thousands of documents the use of phrasal terms is not just desirable, it becomes necessary. An accurate syntactic analysis is an essential prerequisite for selec- tion of phrasal terms. Various statistical methods, e.g., based on word co-occurrences and mutual information, as well as partial parsing techniques, are prone to high error rates (sometimes as high as 50%), turning out many unwanted associations. Therefore a good, fast parser is necessary, but it is by …

Content truncated to 500 words