Inferring Which Medical Treatments Work from Reports of Clinical Trials

Proceedings of NAACL-HLT 2019, pages 3705–3717 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3705 Inferring Which Medical Treatments Work from Reports of Clinical Trials Eric Lehman Northeastern University lehman.e@northeastern.edu Jay B. DeYoung Northeastern University deyoung.j@northeastern.edu Regina Barzilay MIT regina@csail.mit.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Abstract How do we know if a particular medical treat- ment actually works? Ideally one would con- sult all available evidence from relevant clini- cal trials. Unfortunately, such results are pri- marily disseminated in natural language scien- tific articles, imposing substantial burden on those trying to make sense of them. In this pa- per, we present a new task and corpus for mak- ing this unstructured evidence actionable. The task entails inferring reported findings from a full-text article describing a randomized con- trolled trial (RCT) with respect to a given in- tervention, comparator, and outcome of inter- est, e.g., inferring if an article provides evi- dence supporting the use of aspirin to reduce risk of stroke, as compared to placebo. We present a new corpus for this task com- prising 10,000+ prompts coupled with full- text articles describing RCTs. Results using a suite of models — ranging from heuris- tic (rule-based) approaches to attentive neu- ral architectures — demonstrate the difficulty of the task, which we believe largely owes to the lengthy, technical input texts. To facili- tate further work on this important, challeng- ing problem we make the corpus, documenta- tion, a website and leaderboard, and code for baselines and evaluation available at http: //evidence-inference.ebm-nlp.com/. 1 Introduction Biomedical evidence is predominantly dissemi- nated in unstructured, natural language scientific manuscripts that describe the conduct and results of randomized control trials (RCTs). The pub- lished evidence base is vast and expanding (Bas- tian et al., 2010): at present more than 100 reports of RCTs are published every day, on average. It is thus time-consuming, and often practically im- possible, to sort through all of the relevant pub- lished literature to robustly answer questions such With respect to <outcome>, what is the reported difference between patients receiving <A> and those receiving <B>? Significantly increased Significantly decreased No significant difference “Patients receiving A experienced significantly more outcome …” article and prompt answer and rationale Figure 1: The task. Given a treatment A, a comparator B, and an outcome, infer the reported relationship be- tween A and B with respect to outcome, and provide evidence supporting this from the text. as: Does infliximab reduce dysmenorrhea (pain) scores, relative to placebo? Given the critical role published reports of tri- als play in informing evidence-based care, orga- nizations such as the Cochrane collaboration and groups at evidence-based practice centers (EPCs) are dedicated to manually synthesizing findings, but struggle to keep up with the literature (Tsaf- nat et al., 2013). NLP can play a key role in automating this process, thereby mitigating costs and keeping treatment recommendations up-to- date with the evidence as it is published. In this paper, we consider the task of inferring whether …

Content truncated to 500 words