Novelty Detection: The TREC Experience

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 105–112, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Novelty Detection: The TREC Experience Ian Soboroff and Donna Harman National Institute of Standards and Technology Gaithersburg, MD (ian.soboroff,donna.harman)@nist.gov Abstract A challenge for search systems is to de- tect not only when an item is relevant to the user’s information need, but also when it contains something new which the user has not seen before. In the TREC novelty track, the task was to highlight sentences containing relevant and new information in a short, topical document stream. This is analogous to highlighting key parts of a document for another person to read, and this kind of output can be useful as input to a summarization system. Search topics in- volved both news events and reported opin- ions on hot-button subjects. When peo- ple performed this task, they tended to se- lect small blocks of consecutive sentences, whereas current systems identified many relevant and novel passages. We also found that opinions are much harder to track than events. 1 Introduction The problem of novelty detection has long been a sig- nificant one for retrieval systems. The “selective dis- semination of information” (SDI) paradigm assumed that the people wanted to be able to track new in- formation relating to known topics as their primary search task. While most SDI and information filter- ing systems have focused on similarity to a topical profile (Robertson, 2002) or to a community of users with a shared interest (Belkin and Croft, 1992), re- cent efforts (Carbonell and Goldstein, 1998; Allan et al., 2000; Kumaran et al., 2003) have looked at the retrieval of specifically novel information. The TREC novelty track experiments were con- ducted from 2002 to 2004 (Harman, 2002; Soboroff and Harman, 2003; Soboroff, 2004). The basic task was defined as follows: given a topic and an ordered set of documents related to that topic, segmented into sentences, return those sentences that are both relevant to the topic and novel given what has al- ready been seen previously in that document set. This task models an application where a user is skim- ming a set of documents, and the system highlights new, on-topic information. There are two problems that participants must solve in this task. The first is identifying relevant sentences, which is essentially a passage retrieval task. Sentence retrieval differs from document re- trieval because there is much less text to work with, and identifying a relevant sentence may involve ex- amining the sentence in the context of those sur- rounding it. The sentence was specified as the unit of retrieval in order to standardize the task across a variety of passage retrieval approaches, as well as to simplify the evaluation. The second problem is that of identifying those rel- evant sentences that contain new information. The operational definition of “new” here is information that has not appeared previously in this topic’s set of documents. In other words, …

Content truncated to 500 words