Overview of TREC-1

OVERVIEW OF TREC-1 Donna Harman National Institute of Standards and Technology Gaithersburg, Md. 20899 ABSTRACT The first Text REtrieval Conference (TREC-I) was held in early November 1992 and was attended by about 100 people working in the 25 participating groups. The goal of the conference was to bring research groups together to discuss their work on a new large test collec- tion. There was a large variety of retrieval techniques reported on, including methods using automatic thesaurii, sophisticated term weighting, natural language tech- niques, relevance feedback, and advanced pattern match- ing. As results had been run through a common evalua- tion package, groups were able to compare the effectiveness of different techniques, and discuss how differences among the systems affected performance. 1. INTRODUCTION There is a long history of experimentation in infor- mation retrieval. Research started with experiments in indexing languages, such as the Cranfield I tests [1], and has continued with over 30 years of experimentation with the retrieval engines themselves. The Cranfield II studies [2] showed that automatic indexing was compar- able to manual indexing, and this and the availability of computers created a major interest in the automatic indexing and searching of texts. The Cranfield experi- ments also emphasized the importance of creating test collections and using these for comparative evaluation. The Cranfield collection, created in the late 1960's, con- tained 1400 documents and 225 queries, and has been heavily used by researchers since then. Subsequently other collections have been built, such as the CACM collection [3], and the NPL collection [4]. In the thirty or so years of experimentation there have been two missing elements. First, although some research groups have used the same collections, there has been no concerted effort by groups to work with the same data, use the same evaluation techniques, and gen- erally compare results across systems. The importance of this is not to show any system to be superior, but to allow comparison across a very wide variety of tech- niques, much wider than only one research group would tackle. Karen Sparck Jones in 1981 [5] commented that: Yet the most slriking feature of the test history of the past two decades is its lack of consolidation. It is true that some very broad generalizations have been endorsed by succes- sive tests: for example...but there has been a real failure at the detailed level to build one test on another. As a result there are no expla- nations for these generalizations, and hence no means of knowing whether improved systems could be designed (p. 245). This consolidation is more likely ff groups can compare results across the same data, using the same evaluation method, and then meet to discuss openly how methods differ. The second missing element, which has become criti- cal in the last ten years, is the lack of a realistically- sized test collection. Evaluation using the small collec- tions currently available may not reflect performance of systems in large full-text searching, and certainly does not demonstrate any proven …

Content truncated to 500 words