Finding content-bearing terms using term similarities

Proceedings of EACL '99 Finding content-bearing terms using term similarities Justin Picard Institut Interfacultaire d'Informatique University of Neuchttel SWITZERLAND justin.picard@seco.unine.ch Abstract This paper explores the issue of using dif- ferent co-occurrence similarities between terms for separating query terms that are useful for retrieval from those that are harmful. The hypothesis under examina- tion is that useful terms tend to be more similar to each other than to other query terms. Preliminary experiments with similarities computed using first-order and second-order co-occurrence seem to confirm the hypothesis. Term similari- ties could then be used for determining which query terms are useful and best reflect the user's information need. A possible application would be to use this source of evidence for tuning the weights of the query terms. 1 Introduction Co-occurrence information, whether it is used for expanding automatically the original query (Qiu and Frei, 1993), for providing a list of candi- date terms to the user in interactive query ex- pansion, or for relaxing the independence as- sumption between query terms (van Rijsbergen, 1977), has been widely used in information re- trieval. Nevertheless, the use of this information has often resulted in reduction of retrieval effec- tiveness (Smeaton and van Rijsbergen, 1983), a fact sometimes explained by the poor discriminat- ing power of the relationships (Peat and Willet, 1991). It was not until recently that a more elabo- rated use of this information resulted in consistent improvement of retrieval effectiveness. Improve- ments came from a different computation of the relationships named "second-order co-occurrence" (Schutze and Pedersen, 1997), from an adequate combination with other sources of evidence such as relevance feedback (Xu and Croft, 1996), or from a more careful use of the similarities for ex- panding the query (Qiu and Frei, 1993). Indeed, interesting patterns relying in co- occurrence information may be discovered and, if used carefully, may enhance retrieval effective- ness. This paper explores the use of co-occurrence similarities between query terms for determining the subset of query terms which are good descrip- tors of the user's information need. Query terms can be divided into those that are useful for re- trieval and those that are harmful, which will be named respectively "content" terms and "noisy" terms. The hypothesis under examination is that two content terms tend to be more similar to each other than would be two noisy terms, or a noisy and a content term. Intuitively, the query terms which reflect the user's information need are more likely to be found in relevant documents and should concern similar topic areas. Consequently, they should be found in similar contexts in the corpus. A similarity measures the degree to which two terms can be found in the same context, and should be higher for two content terms. We name this hypothesis the "Cluster Hypoth- esis for query terms", due to its correspondence with the Cluster Hypothesis of information re- trieval which assumes that relevant documents "are more like one another than they are like non- relevant documents" (van Rijsbergen and Sparck- …

Content truncated to 500 words