Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4474–4486 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4474 Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset Xiang Yue Bernal Jimenez Gutierrez The Ohio State University {yue.149, jimenezgutierrez.1, sun.397}@osu.edu Huan Sun Abstract Machine reading comprehension has made great progress in recent years owing to large- scale annotated datasets. In the clinical do- main, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (2018) tackled this issue by using expert-annotated question templates and existing i2b2 annota- tions to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in- depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative ex- periments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance com- pared to the model trained on the entire dataset, (iv) this performance is close to human ex- pert’s performance, and (v) BERT models do not beat the best performing base model. Fol- lowing our analysis of the emrQA, we fur- ther explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen ques- tions and contexts. We argue that both should be considered when creating future datasets.1 1 Introduction Medical professionals often query over clinical notes in Electronic Medical Records (EMRs) to find information that can support their decision making (Demner-Fushman et al., 2009; Rosen- bloom et al., 2011; Wang et al., 2018). One way to facilitate such information seeking activities is to build a natural language question answering (QA) system that can extract precise answers from clin- ical notes (Cairns et al., 2011; Cao et al., 2011; Wren, 2011; Abacha and Demner-Fushman, 2016, 2019). 1Our code is available at https://github.com/ xiangyue9607/CliniRC. Context:�...�For�HTN�control,�pt�was�given�HCTZ and�lopressor�which�sufficiently�controlled�his�BP.�Pt was�sent�home�on�HCTZ�25mg�daily�and�atenolol ...50mg�daily.� ADDITIONAL�COMMENTS:�1.)�Take�hydrochlo- rothiazide�25mg�daily�and�atenolol�50mg�daily�for your�blood�pressure.�You�should�also�take�aspirin 81mg�daily.� Question:�What�was�the�dosage�prescribed�of� hydrochlorothiazide? Answer:�ADDITIONAL�COMMENTS:�1.)�Take hydrochlorothiazide�25mg�daily�and�atenolol�50mg daily�for�your RECORD�#992321,�Date:�2145-09-22 Question:�Why�has�the�patient�been�prescribed�hctz? Answer:�For�HTN�control,�pt�was�given�HCTZ� and�lopressor�which�sufficiently Figure 1: Examples from the emrQA dataset: Part of a clinical note as context and 2 question-answer pairs. Due to the original emrQA generation issues, often- times answers are incomplete or contain irrelevant parts to the questions (the underlined parts are what we think the most relevant to the questions). Machine reading comprehension (RC) aims to automatically answer questions based on a given document or text corpus and has drawn wide atten- tion in recent years. Many neural models (Cheng et al., 2016; Wang et al., 2017; Wang and Jiang, 2017; Seo et al., 2017; Chen et al., 2017; Devlin et al., 2019) have achieved very promising results on this task, owing to large-scale QA datasets (Hermann et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Yang et al., 2018). Unfortunately, clinical …

Content truncated to 500 words