A Machine Learning based Approach to Evaluating Retrieval Systems

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 399–406, New York, June 2006. c⃝2006 Association for Computational Linguistics A Machine Learning based Approach to Evaluating Retrieval Systems Huyen-Trang Vu and Patrick Gallinari Laboratory of Computer Science (LIP6) University of Pierre and Marie Curie 8 rue du capitaine Scott, 75015 Paris, France {vu,gallinar}@poleia.lip6.fr Abstract Test collections are essential to evaluate Information Retrieval (IR) systems. The relevance assessment set has been recog- nized as the key bottleneck in test col- lection building, especially on very large sized document collections. This paper addresses the problem of efficiently se- lecting documents to be included in the assessment set. We will show how ma- chine learning techniques can fit this task. This leads to smaller pools than tradi- tional round robin pooling, thus reduces significantly the manual assessment work- load. Experimental results on TREC col- lections1 consistently demonstrate the ef- fectiveness of our approach according to different evaluation criteria. 1 Introduction The effectiveness of retrieval systems is often justi- fied by benchmark test collections. A standard test collection consists of lots of documents, a set of in- formation needs, called topics and human judgment about the relevance status of each document for a topic. Nowadays, it is relatively easy to gather huge set of millions of documents and hundreds of topics. The key obstacle for forming large sized test col- lections lies therefore in the topic assessment pro- cedure. Assessing the whole document sets is un- feasible, even for small sized collection of 800,000 documents (Voorhees and Harman, 1999). In order to keep the assessment process practical, one often 1http://trec.nist.gov selects a certain number of documents for judgment. This is called (document) pooling and the outcome the pool or the qrels (query relevance set). The col- lected documents are then judged by humans, doc- uments outside the pool are assumed non relevant. A representative pool is therefore essential to the whole evaluation process. This paper proposes a method to form the assess- ment set with the support of machine learning algo- rithms. Based on relevance judgments of relatively shallow pools, a ranking algorithm will attempt to give priority for relevant documents so that the as- sessment set can be fixed at a feasible size without skewing the system evaluation result. The judgment process is indeed kept as much subjective-free as possible: the first relevance feeback step is designed appropriately so that the assessor cannot give any bias towards any particular rank or any system, the learning process is completely transparent to the as- sessors and parameters of the ranking function are collection-tailored rather than exported from previ- ous collections. The method will then be evaluated on TREC ad-hoc collections. Results from our comprehensive experiment confirm that the qrels generated by our method are much more representative than those of the same size by the TREC method. The outcome qrels is substantially smaller, so much cheaper to produce than the official TREC qrels, yet their con- clusions about …

Content truncated to 500 words