BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR

Proceedings of the 28th International Conference on Computational Linguistics, pages 669–679 Barcelona, Spain (Online), December 8-13, 2020 669 BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR Souradip Chakraborty∗ Walmart Labs souradip24@gmail.com Shweta Bhatt∗ Jupiter shwetabhatt08@gmail.com Francesco Mosconi Zero to Deep Learning f@mosconi.me Ekaba Bisong∗ SiliconBlast ekaba.bisong@siliconblast.com Thomas O. Wagner III TPHS Research thomas.wagner@bertbiomed.org Riley D. Elliott TPHS Research riley.elliott@bertbiomed.org Abstract The SARS-CoV-2 (COVID-19) pandemic spotlighted the importance of moving quickly with biomedical research. However, as the number of biomedical research papers continue to in- crease, the task of finding relevant articles to answer pressing questions has become significant. In this work, we propose a textual data mining tool that supports literature search to accelerate the work of researchers in the biomedical domain. We achieve this by building BioMedBERT, a neural-based deep contextual understanding model for Question-Answering (QA) and Infor- mation Retrieval tasks. We also leverage the new BREATHE dataset which is one of the largest available datasets of biomedical research literature, containing abstracts and full-text articles from ten different biomedical literature sources on which we pre-train our BioMedBERT model. Our work achieves state-of-the-art results on the QA fine-tuning task on BioASQ 5b, 6b and 7b datasets. In addition, we observe superior relevant results when BioMedBERT embeddings are used with Elasticsearch for the Information Retrieval task on the intelligently formulated BioASQ dataset. We believe our diverse dataset and our unique model architecture are what led us to achieve the state-of-the-art results for QA and IR tasks. 1 Introduction The COVID-19 pandemic reminded us of the need for a tool that biomedical researchers can use to sift through existing research to extract novel insights, and ultimately help them make novel drug discov- eries. The rate of new publications in the biomedical field is on the rise. PubMed reports that more than 1 million biomedical research papers are published each year, amounting to nearly two papers per minute (Landhuis, 2016). For papers mentioning COVID-19 alone, as of June 2020 more than 8000 peer- reviewed publications have been published on PubMed. With the rate of scientific papers on COVID-19 doubling every fourteen days (Coren, 2020), it is imperative to have a language understanding tool that can extract relevant information from credible literature, such as the research methodology, data, authors, results, and citations (Hao, 2020). In this paper, we address the problem from an information retrieval perspective, extracting the textual and contextual information from the corpus by taking a hierarchical approach. Traditional search ap- proaches such as Lucene-based Elasticsearch (Gormley and Tong, 2015) using BM-25 & Jaccard-based matrices are efficient in retrieving objective answers where the primary task is to extract specific parts of ∗ Denotes first author with equal contribution. Github repo: https://github.com/BioMedBERT/biomedbert. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 670 the passage. However such methods struggle in the contextual retrieval of documents for which we need latent space representation of the query and the corpus of passages. Our work leverages the BERT …

Content truncated to 500 words