Creating a Test Collection for Citation-based IR Experiments

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 391–398, New York, June 2006. c⃝2006 Association for Computational Linguistics Creating a Test Collection for Citation-based IR Experiments Anna Ritchie University of Cambridge Computer Laboratory 15 J J Thompson Avenue Cambridge, CB3 0FD, U.K. ar283@cl.cam.ac.uk Simone Teufel University of Cambridge Computer Laboratory 15 J J Thompson Avenue Cambridge, CB3 0FD, U.K. sht25@cl.cam.ac.uk Stephen Robertson Microsoft Research Ltd Roger Needham House 7 J J Thomson Avenue Cambridge, CB3 0FB, U.K. ser@microsoft.com Abstract We present an approach to building a test collection of research papers. The ap- proach is based on the Cranfield 2 tests but uses as its vehicle a current conference; research questions and relevance judge- ments of all cited papers are elicited from conference authors. The resultant test col- lection is different from TREC’s in that it comprises scientific articles rather than newspaper text and, thus, allows for IR experiments that include citation informa- tion. The test collection currently con- sists of 170 queries with relevance judge- ments; the document collection is the ACL Anthology. We describe properties of our queries and relevance judgements, and demonstrate the use of the test collection in an experimental setup. One potentially problematic property of our collection is that queries have a low number of relevant documents; we discuss ways of alleviating this. 1 Introduction We present a methodology for creating a test collec- tion of scientific papers that is based on the Cran- field 2 methodology but uses a current conference as the main vehicle for eliciting relevance judgements from users, i.e., the authors. Building a test collection is a long and expensive process but was necessary as no ready-made test col- lection existed on which the kinds of experiments with citation information that we envisage could be run. We aim to improve term-based IR on scien- tific articles with citation information, by using in- dex terms from the citing article to additionally de- scribe the cited document. Exactly how to do this is the research question that our test collection should help to address. This paper is structured as follows: Section 2 mo- tivates our proposed experiments and, thereby, our test collection. Section 3 discusses the how test col- lections are built and, in particular, our own. Sec- tion 4 briefly describes the practicalities of compil- ing the document collection and the processing we perform to prepare the documents for our experi- ments. In Section 5, we show that our test collection can be used with standard IR tools. Finally, Sec- tion 6 discusses the problem of the low number of relevant documents judged so far and two ways of alleviating this problem. 2 Motivation The idea of using terms external to a document, coming from a ‘citing’ document, has been bor- rowed from web-based IR. When one paper cites another, a link is made between them and this link structure is analogous to that of the web: “hyper- links ... provide semantic linkages between …

Content truncated to 500 words