SuMe: A Dataset Towards Summarizing Biomedical Mechanisms

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6922–6931 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6922 SuMe: A Dataset Towards Summarizing Biomedical Mechanisms Mohaddeseh Bastan♠, Nishant Shankar♣, Mihai Surdeanu♡, Niranjan Balasubramanian♠ ♠ Stony Brook University ♣Delft University of Technology ♡ University of Arizona {mbastan, niranjan}@cs.stonybrook.edu n.shankar@tudelft.nl msurdeanu@email.arizona.edu Abstract Can language models read biomedical texts and explain the biomedical mechanisms discussed? In this work we introduce a biomedical mechanism summarization task. Biomedical studies often investigate the mechanisms behind how one entity (e.g., a protein or a chemical) affects another in a biological context. The abstracts of these publications often include a focused set of sentences that present relevant supporting statements regarding such relationships, associated experimental evidence, and a concluding sentence that summarizes the mechanism underlying the relationship. We leverage this structure and create a summarization task, where the input is a collection of sentences and the main entities in an abstract, and the output includes the relationship and a sentence that summarizes the mechanism. Using a small amount of manually labeled mechanism sentences, we train a mechanism sentence classifier to filter a large biomedical abstract collection and create a summarization dataset with 22k instances. We also introduce conclusion sentence generation as a pretraining task with 611k instances. We benchmark the performance of large bio-domain language models. We find that while the pretraining task help improves performance, the best model produces acceptable mechanism outputs in only 32% of the instances, which shows the task presents significant challenges in biomedical language understanding and summarization. 1. Keywords: Explanation Generation, Text Generation, Summarization, Biomedical NLP, Relation Extraction 1. Introduction Understanding biochemical mechanisms such as pro- tein signaling pathways is one of the central pursuits of biomedical research (Str¨otgen and Gertz, 2012) (Arighi et al., 2011; Krallinger et al., 2017; Demner-Fushman et al., 2020). Biomedical research has advanced tremen- dously in the past few decades, to the point where we now suffer from “an embarrassment of riches”. Publica- tions are generated at such a rapid pace (PubMed 2 has indexed more than 1 million publications per year in the past 8 years!) that we need information access applica- tions which can help extract and organize biomedical relations and summarize the biomedical mechanisms underlying them. Developing models that can read biomedical texts and reason about these mechanisms is an important step towards this. In this paper, we introduce a mechanism summariza- tion task which couples text that discusses elements of biomedical mechanisms with their summaries. The task requires models to read text that presents information about the connection between two target entities and generate a summary sentence that explains the under- lying mechanism and the relation between the entities. We see this task from two perspectives. First, summa- rizing biomedical mechanisms can be seen as part of the broader efforts in extracting (Czarnecki et al., 2012), organizing (Kemper et al., 2010; Kemper et al., 2010; Miwa et al., 2013; Subramani et al., 2015; Poon et …

Content truncated to 500 words