On the Use of Term Associations in Automatic Information Retrieval

On the Use of Term Associations in Automatic Information Retrieval Gerard Salton* Abstract It has been recognized that single words extracted from natural language texts are not always useful for the representation of information content. Associated or related terms, and complex content identifiers derived from thesauruses and knowledge bases, or constructed by automatic word grouping techniques, have therefore been proposed for text identification purposes. The area of associative content analysis and information retrieval is reviewed in this study. The available experimental evidence shows that none of the existing or proposed methodologies are guaranteed to improve retrieval performance in a replicable manner for document collections in different subject areas. The associative techniques are most valu- able for restricted environments covering narrow subject areas, or in iterative search situa- tions where user inputs are available to refine previously available query formulations and search output. I. Introduction Computers were first used for the process- ing of natural language texts over 30 years ago. From the beginning it has been recognized that the individual words contained in the texts of written documents could be used in part to pro- vide a representation of document content. At the same time it was generally accepted that certain words, or word sets, would not produce meaningful content identifiers. In particular, some quite broad words, such as the term "com- puter" used to identify computer science litera- ture, would be useless for distinguishing one document from another. Other very specific terms would be so rare that no single item in a collection might reasonably be described by such a very rare term. To improve the operations of text process- ing systems, it has been suggested that the ori- ginal document vocabulary be expanded by adding related or associated terms not originally present in the available text samples. Two main types of vocabulary relationships can be recog- nized in this connection, known respectively as and ~ relations. [I] The paradigmetric relations cover term associa- tions, such as synonyms and hierarchical inclu- sion, that always exist between particular terms regardless of the context in which these terms are used. For example, a paradigmatic relation exists between the name of a country (say, France) and the capital city (Paris). Syntag.- matie relations, on the other hand, are rela- tions which are not valid outside some specified context. For example, a cause-effect relation may be detected in certain circumstances between "poison" and "death H. Department of Computer Science, Cornell Univer- sity, Ithaca, NY 14853. This study was supported in part by the National Science Foundation under grants IST 83-16166 and IST 85-44189, 380 The paradigmatic relations may be identi- fied by using precoustructed dictionaries, or thesauruses, containing schedules or groupings of related terms or concepts. The syntagmatic relations, on the other hand, must be derived by analyzing particular text samples and extracting the term relationships specified in these texts. Various methods are outlined in the next section for utilizing paradigmatic and syntag- matic term associations in text processing sys- tems, and the …

Content truncated to 500 words