BioRead: A New Dataset for Biomedical Reading Comprehension

BioRead: A New Dataset for Biomedical Reading Comprehension Dimitris Pappas1,2, Ion Androutsopoulos1, Haris Papageorgiou2 1Department of Informatics, Athens University of Economics and Business, Patission 76, GR-104 34 Athens, Greece 2Institute for Language and Speech Processing, ATHENA Research Center, Epidavrou & Artemidos 6, GR-151 25 Maroussi, Greece pappasd@aueb.gr, ion@aueb.gr, xaris@ilsp.gr Abstract We present BioRead, a new publicly available cloze-style biomedical machine reading comprehension (MRC) dataset with approxi- mately 16.4 million passage-question instances. BioRead was constructed in the same way as the widely used Children’s Book Test and its extension BookTest, but using biomedical journal articles and employing MetaMap to identify UMLS concepts. BioRead is one of the largest MRC datasets, and currently the largest one in the biomedical domain. We also provide a subset of BioRead, BioReadLite, for research groups with fewer computational resources. We re-implemented and tested on BioReadLite two well-known MRC methods, AS Reader and AOA Reader, along with four baselines, as a first step towards a BioRead (and BioReadLite) leaderboard. AOA Reader is currently the best method on BioReadLite, with 51.19% test accuracy. Both AOA Reader and AS Reader outperform the baselines by a wide margin on the test subset of BioReadLite. Our re-implementations of the two MRC methods are also publicly available. Keywords: BioRead, biomedical, dataset, corpus, evaluation, reading comprehension, question answering, deep learning. 1. Introduction Machine Reading Comprehension (MRC) systems (Her- mann et al., 2015) are given a passage (e.g., from a news ar- ticle or book) and they are required to answer a question by considering the information of the passage. Manually con- structing MRC datasets is very labour-intensive and leads to datasets that may not be large enough to train data-hungry deep learning methods (Goodfellow et al., 2016; Goldberg, 2017). For example, the BioASQ dataset (Tsatsaronis et al., 2015), which was constructed by biomedical experts, cur- rently contains only two thousand questions approximately. The largest manually curated MRC dataset we are aware of, SQuAD (Rajpurkar et al., 2016), which was crowdsourced, comprises approximately 100k passage-question instances. Larger datasets can be constructed automatically by con- sidering only ‘cloze-style’ questions, which require filling in a missing word or phrase in a given sentence about the passage (e.g., “ has been implicated in the pathogenesis of PD.”). For example, CBTest (Hill et al., 2015) contains passages from children’s books; each cloze-style question is a sentence that follows the corresponding passage in its book, with a randomly selected common noun, named en- tity, verb, or preposition of the sentence removed and turned into a slot to be filled in. CBTest contains approximately 687k passage-question instances. It was more recently ex- panded to BookTest (Bajgar et al., 2016), which comprises approximately 14 million passage-question instances, by applying the same methodology to a much larger collec- tion of books. The CNN and Daily Mail datasets (Hermann et al., 2015) were produced in a similar manner. They com- prise news articles and cloze-style questions constructed by removing words from sentences summarising the articles; they contain approx. 380k and 880k instances, respectively. …

Content truncated to 500 words