PubMedQA: A Dataset for Biomedical Research Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2567–2577, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2567 PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin University of Pittsburgh qiao.jin@pitt.edu Bhuwan Dhingra Carnegie Mellon University bdhingra@cs.cmu.edu Zhengping Liu University of Pittsburgh zliu@pitt.edu William W. Cohen Google AI wcohen@google.com Xinghua Lu University of Pittsburgh xinghua@pitt.edu Abstract We introduce PubMedQA, a novel biomedi- cal question answering (QA) dataset collected from PubMed abstracts. The task of Pub- MedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding ab- stracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially gen- erated QA instances. Each PubMedQA in- stance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the re- search question, and (4) a yes/no/maybe an- swer which summarizes the conclusion. Pub- MedQA is the first QA dataset where rea- soning over biomedical research texts, espe- cially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accu- racy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for im- provement. PubMedQA is publicly available at https://pubmedqa.github.io. 1 Introduction A long-term goal of natural language understand- ing is to build intelligent systems that can reason and infer over natural language. The question an- swering (QA) task, in which models learn how to answer questions, is often used as a benchmark for quantitatively measuring the reasoning and infer- ring abilities of such intelligent systems. While many large-scale annotated general do- main QA datasets have been introduced (Ra- jpurkar et al., 2016; Lai et al., 2017; Koˇcisk`y Question: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting? Context: (Objective) Recent studies have demonstrated that statins have pleiotropic effects, including anti-inflammatory ef- fects and atrial fibrillation (AF) preventive effects [...] (Methods) 221 patients underwent CABG in our hospital from 2004 to 2007. 14 patients with preoperative AF and 4 patients with concomitant valve surgery [...] (Results) The overall incidence of postoperative AF was 26%. Postoperative AF was significantly lower in the Statin group compared with the Non-statin group (16% versus 33%, p=0.005). Multivariate analysis demon- strated that independent predictors of AF [...] Long Answer: (Conclusion) Our study indicated that preoperative statin therapy seems to reduce AF development after CABG. Answer: yes Figure 1: An instance (Sakamoto et al., 2011) of Pub- MedQA dataset: Question is the original question title; Context includes the structured abstract except its con- clusive part, which serves as the Long Answer; Human experts annotated the Answer yes. Supporting fact for the …

Content truncated to 500 words