MSˆ2: Multi-Document Summarization of Medical Studies

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7494–7513 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7494 MSˆ2: A Dataset for Multi-Document Summarization of Medical Studies Jay DeYoung1∗ Iz Beltagy2 Madeleine van Zuylen2 Bailey Kuehl2 Lucy Lu Wang2 Northeastern University1 Allen Institute for AI2 deyoung.j@northeastern.edu {beltagy,madeleinev,baileyk,lucyw}@allenai.org Abstract To assess the effectiveness of any medical in- tervention, researchers must conduct a time- intensive and manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MSˆ2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20K summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory ev- idence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical do- main. We experiment with a summarization system based on BART, with promising early results, though significant work remains to achieve higher summarization quality. We for- mulate our summarization inputs and targets in both free text and structured forms and mod- ify a recently proposed metric to assess the quality of our system’s generated summaries. Data and models are available at https:// github.com/allenai/ms2. 1 Introduction Multi-document summarization (MDS) is a chal- lenging task, with relatively limited resources and modeling techniques. Existing datasets are either in the general domain, such as WikiSum (Liu et al., 2018) and Multi-News (Fabbri et al., 2019), or very small such as DUC1 or TAC 2011 (Owczarzak and Dang, 2011). In this work, we add to this burgeon- ing area by developing a dataset for summarizing biomedical findings. We derive documents and summaries from systematic literature reviews, a type of biomedical paper that synthesizes results across many other studies. Our aim in introducing MSˆ2 is to: (1) expand MDS to the biomedical domain, (2) investigate fundamentally challenging ∗Work performed during internship at AI2 1https://duc.nist.gov Figure 1: Our primary formulation (texts-to-text) is a seq2seq MDS task. Given study abstracts and a BACK- GROUND statement, generate the TARGET summary. Figure 2: The distributions of review and study publica- tion years in MSˆ2 show a clear temporal lag. Dashed lines mark the median year of publication. issues in NLP over scientific text, such as summa- rization over contradictory information and assess- ing summary quality via a structured intermediate form, and (3) aid in distilling large amounts of biomedical literature by supporting automated gen- eration of literature review summaries. Systematic reviews synthesize knowledge across many studies (Khan et al., 2003), and they are so called for the systematic (and expensive) process of creating a review; each taking 1-2 years to com- plete (Michelson and Reuter, 2019).2 As we note 2https://community.cochrane.org/review-production/ production-resources/proposing-and-registering-new- cochrane-reviews 7495 in Fig. 2, a delay of around 8 years is observed between reviews and the studies they cite! The time and cost of creating and updating reviews has inspired efforts at automation (Tsafnat et al., 2014; Marshall et al., 2016; Beller et al., …

Content truncated to 500 words