Modelling Stopping Criteria for Search Results using Poisson Processes

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3484–3489, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3484 Modelling Stopping Criteria for Search Results using Poisson Processes Alison Sneyd and Mark Stevenson Natural Language Processing Group Department of Computer Science University of Sheffield a.sneyd,mark.stevenson@sheffield.ac.uk Abstract Text retrieval systems often return large sets of documents, particularly when applied to large collections. Stopping criteria can reduce the number of these documents that need to be manually evaluated for relevance by predict- ing when a suitable level of recall has been achieved. In this work, a novel method for de- termining a stopping criterion is proposed that models the rate at which relevant documents occur using a Poisson process. This method allows a user to specify both a minimum de- sired level of recall to achieve and a desired probability of having achieved it. We evaluate our method on a public dataset and compare it with previous techniques for determining stop- ping criteria. 1 Introduction Document retrieval systems are widely used in in- formation retrieval (IR) to return a set of docu- ments in response to a user query. They often re- turn large numbers of documents, particularly for large collections, which places a significant bur- den on a user who has to manually review them to determine which are relevant (i.e. of interest). The problem is particularly acute in scenarios where relevant documents are rare in the set of retrieved documents, or where there is a high cost associ- ated with missing a relevant document. Systematic reviews of medical literature are a good example of such a task. They aim to identify and summarise all available evidence for a spe- cific research question such as ‘Is systemic inflam- mation present in stable chronic obstructive pul- monary disease?’(Gan et al., 2004). Evidence in- cluded in systematic reviews is typically identified by executing a Boolean query optimised for high recall, with the results manually reviewed by do- main experts. However, this process is extremely expensive and a single review may cost up to a quarter of a million dollars and take over a year to complete (McGowan and Sampson, 2005; Karimi et al., 2010). A range of studies have demonstrated that ranking the results of the Boolean query can support the identification of relevant studies by placing them higher in the ranking, e.g. (Miwa et al., 2014; Wallace et al., 2010; O’Mara-Eves et al., 2015; Shemilt et al., 2016). However, rank- ing alone does not reduce the manual review work- load since it does not inform reviewers about when they can stop looking at search results. Search stopping criteria are methods for deter- mining when manual examination of search re- sults can stop. They essentially provide informa- tion about the risk of missing relevant documents that is associated with not examining some por- tion of the documents returned by a query. Stop- ping criteria have been …

Content truncated to 500 words