RadQA: A Question Answering Dataset to Improve Comprehension of Radiology Reports

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6250–6259 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6250 RadQA: A Question Answering Dataset to Improve Comprehension of Radiology Reports Sarvesh Soni1, Meghana Gudala, Atieh Pajouhi, Kirk Roberts2 School of Biomedical Informatics, The University of Texas Health Science Center at Houston {1sarvesh.soni, 2kirk.roberts}@uth.tmc.edu Abstract We present a radiology question answering dataset, RadQA, with 3074 questions posed against radiology reports and annotated with their corresponding answer spans (resulting in a total of 6148 question-answer evidence pairs) by physicians. The questions are manually created using the clinical referral section of the reports that take into account the actual information needs of ordering physicians and eliminate bias from seeing the answer context (and, further, organically create unanswerable questions). The answer spans are marked within the Findings and Impressions sections of a report. The dataset aims to satisfy the complex clinical requirements by including complete (yet concise) answer phrases (which are not just entities) that can span multiple lines. We conduct a thorough analysis of the proposed dataset by examining the broad categories of disagreement in annotation (providing insights on the errors made by humans) and the reasoning requirements to answer a question (uncovering the huge dependence on medical knowledge for answering the questions). The advanced transformer language models achieve the best F1 score of 63.55 on the test set, however, the best human performance is 90.31 (with an average of 84.52). This demonstrates the challenging nature of RadQA that leaves ample scope for future method research. Keywords: question answering, machine reading comprehension, radiology reports, clinical notes 1. Introduction Question answering (QA) is an intuitive means to query text data and it is especially helpful in the case of large and complex documents. Machine reading comprehen- sion (MRC) has been widely explored to this end in or- der to better comprehend unstructured text, by enabling machines to answer specific questions given a textual passage (Zeng et al., 2020). Much of these pursuits are powered by neural models and since a well-constructed dataset is pivotal to building a suitable model (for a given requirement, domain, or task) there is an explo- sion of MRC datasets in recent years (Dzendzik et al., 2021). However, little work is drawn toward the clini- cal domain to build challenging MRC datasets for im- proving comprehension of the semantically complex and diverse electronic health record (EHR) data. In medicine, much work in MRC is targeted to- ward biomedical scientific articles, which comes under the umbrella of biomedical QA (Athenikos and Han, 2010; Jin et al., 2021). But, owing to the fundamental differences between the text data present in scientific articles and EHRs (Friedman et al., 2002), the datasets (and models) for the former cannot be directly used for the latter. Moreover, between the two types of EHR data, adequately-sized QA datasets are constructed for the structured part (e.g., semantic parsing) (Roberts and Demner-Fushman, 2016; Pampari et al., 2018; Wang et al., …

Content truncated to 500 words