Automating Document Discovery in the Systematic Review Process: How to Use Chaff to Extract Wheat

Automating Document Discovery in the Systematic Review Process: How to Use Chaff to Extract Wheat Christopher Norman,1,2 Mariska Leeflang,2 Pierre Zweigenbaum,1 Aurélie Névéol1 1 LIMSI, CNRS, Université Paris Saclay, F-91405 Orsay 1 Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands {christopher.norman, pierre.zweigenbaum, aurelie.neveol}@limsi.fr m.m.leeflang@amc.uva.nl Abstract Systematic reviews in e.g. empirical medicine address research questions by comprehensively examining the entire published literature. Conventionally, manual literature surveys decide inclusion in two steps, first based on abstracts and title, then by full text, yet current methods to automate the process make no distinction between gold data from these two stages. In this work we compare the impact different schemes for choosing positive and negative examples from the different screening stages have on the training of automated systems. We train a ranker using logistic regression and evaluate it on a new gold standard dataset for clinical NLP, and on an existing gold standard dataset for drug class efficacy. The classification and ranking achieves an average AUC of 0.803 and 0.768 when relying on gold standard decisions based on title and abstracts of articles, and an AUC of 0.625 and 0.839 when relying on gold standard decisions based on full text. Our results suggest that it makes little difference which screening stage the gold standard decisions are drawn from, and that the decisions need not be based on the full text. The results further suggest that common-off-the-shelf algorithms can reduce the amount of work required to retrieve relevant literature. Keywords: Evidence Based Medicine, Information Storage and Retrieval, Review Literature as Topic 1. Introduction Systematic reviews seek to systematically gather all pub- lished evidence addressing a given research question and analyze the aggregate results. Systematic reviews consti- tute some of the strongest forms of scientific evidence, are an integral part of evidence based medicine, and serve a key role in informing and guiding public and institutional decision-making (Wright et al., 2007). One limiting factor of systematic reviews is that they tend to be prohibitively costly to produce.1 The number of ref- erences needed to be manually screened in order to sat- isfy the requirement that virtually all relevant articles have been identified can number in the tens of thousands. Often only some dozens of these references are selected for the final meta-analysis, and the selection process may require months of work for several reviewers (O’Mara-Eves et al., 2015). The screening process starts with identifying an initial set of candidate references, typically by searching databases using boolean queries handcrafted by experts. From this initial set of references, reviewers first screen for inclusion based on titles and abstracts, and then based on the full text (O’Mara-Eves et al., 2015) as illustrated in figure 1. In this paper we will call the references excluded in the first screening stage No (‘N’), references excluded in the sec- ond screening stage Maybe (‘M’), and references included in the final analysis Yes (‘Y’). This selection is divided into two stages because while fi- nal decisions can only be based on the …

Content truncated to 500 words