BioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5770 - 5793 December 7-11, 2022 ©2022 Association for Computational Linguistics BIOREADER: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature Giacomo Frisoni, Miki Mizutani, Gianluca Moro, Lorenzo Valgimigli Department of Computer Science and Engineering (DISI) University of Bologna, Via dell’Università 50, I-47522 Cesena, Italy {giacomo.frisoni, gianluca.moro, lorenzo.valgimigli}@unibo.it {miki.mizutani}@studio.unibo.it Abstract The latest batch of research has equipped lan- guage models with the ability to attend over relevant and factual information from non- parametric external sources, drawing a comple- mentary path to architectural scaling. Besides mastering language, exploiting and contextual- izing the latent world knowledge is crucial in complex domains like biomedicine. However, most works in the field rely on general-purpose models supported by databases like Wikipedia and Books. We introduce BIOREADER1, the first retrieval-enhanced text-to-text model for biomedical natural language processing. Our domain-specific T5-based solution augments the input prompt by fetching and assembling relevant scientific literature chunks from a neu- ral database with ≈60 million tokens centered on PubMed. We fine-tune and evaluate BIORE- ADER on a broad array of downstream tasks, significantly outperforming several state-of-the- art methods despite using up to 3x fewer pa- rameters. In tandem with extensive ablation studies, we show that domain knowledge can be easily altered or supplemented to make the model generate correct predictions bypassing the retraining step and thus addressing the liter- ature overload issue. 1 Introduction In the last decade, deep learning advancements have boosted the development of many solutions for effectively extracting knowledge from biolog- ical data (Domeniconi et al., 2014a, 2016a) and biomedical literature (di Lena et al., 2015)—widely accessible through repositories such as PubMed, PMC, and ScienceDirect. Large pre-trained lan- guage models (PLMs) have become the dominant NLP paradigm, achieving unprecedented results in a panoply of tasks, from named entity recognition (Lee et al., 2020) and semantic parsing (Frisoni et al., 2021, 2022b) to information retrieval (Moro 1https://github.com/disi-unibo-nlp Biomedical Scientific Evidence Indexing Heterogeneity in DNA damage within the cell population was observed as a function of radiation dose Related context genomic instability and mutation Cholestyramine: Concomitant * DRUG * administration decreased the mean AUC of total * DRUG * approximately 55%. DDI-mechanism In which location is dimeric IgA most likely to be found in highest concentrations? Colostrum Query Document Classification Relation Extraction ... Non-knowledge intensive Tasks Open-domain Question Answering ... Knowledge intensive Tasks BioREADER PMID: 12947391 Radiation-induced genomic instability is... Figure 1: Illustration of BIOREADER, a text-to-text model conditioned on scientific evidence retrieved from an explicit PubMed-based datastore. Every biomedical task is cast as translating text spans with the help of external domain knowledge retrieved on the fly. and Valgimigli, 2021) and document summariza- tion (Moro et al., 2022). To justify this success, PLMs have been shown to implicitly hold a substantial amount of in- depth knowledge in their parameters (Petroni et al., 2019; Davison et al., 2019), resulting from self- supervised learning on extreme-scale text corpora. Efforts to this point have mainly focused on predictably improving NLP …

Content truncated to 500 words