The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5098 - 5111 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The patient is more dead than alive: exploring the current state of the multi-document summarization of the biomedical literature Yulia Otmakhova1, Karin Verspoor2, Timothy Baldwin1,3, Jey Han Lau1 1The University of Melbourne, 2RMIT University, 3MBZUAI yotmakhova@student.unimelb.edu.au, karin.verspoor@rmit.edu.au, tb@ldwin.net, jeyhan.lau@gmail.com Abstract Although multi-document summarization (MDS) of the biomedical literature is a highly valuable task that has recently attracted substantial interest, evaluation of the quality of biomedical summaries lacks consistency and transparency. In this paper, using systematic reviews as an example of biomedical MDS, we examine the summaries generated by two current models in order to understand the deficiencies of existing evaluation approaches in the context of the challenges that arise in the MDS task. Based on this analysis, we propose a new approach to human evaluation and identify several challenges that must be overcome to develop effective biomedical MDS systems. 1 Introduction With the number of biomedical publications dou- bling every two years (Cios et al., 2019), it is diffi- cult for medical professionals to incorporate new, often contradictory, evidence into their daily work, as it would require appraising, comparing and syn- thesising the outcomes of multiple primary studies (Sackett and Rosenberg, 1996). Systematic reviews, which aggregate such evidence from multiple clin- ical trials, provide only a partial solution, as they are very time-consuming to write and thus can be unavailable for newer clinical questions or quickly become outdated. In this context, the ability to automatically summarize evidence from multiple studies is of high practical importance. The task, however, is more challenging than general multi- document summarization (MDS), as the summaries must correctly draw conclusions based on often contradictory studies, and aggregate details such as groups of patients or names and doses of treatments, in addition to dealing with often-cited difficulties posed by biomedical text such as complex lexi- cal and semantic relationships between concepts (Plaza et al., 2011). Though recent approaches to biomedical summarization acknowledge the addi- tional challenges of the task and try to incorporate some domain-specific knowledge to deal with them (Wallace et al., 2021; Shah et al., 2021; DeYoung et al., 2021), we still lack a solid understanding of how well current models capture such knowledge, how useful the generated summaries are, or how to measure progress. In this paper, we propose a systematic approach to human evaluation of biomedical summaries, and apply it to analyse the summaries generated by two state-of-art systems. We examine the common er- rors in generated summaries and the correlation of automatic metrics such as ROUGE (Lin, 2004) with our evaluation results. We choose summariza- tion models proposed by DeYoung et al. (2021), as they not only demonstrate the abilities of end- to-end neural models, but also incorporate domain- specific knowledge such as entity prompts. The contributions of this paper are as follows: (1) We propose a new approach to human evalu- ation of …

Content truncated to 500 words