CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension

Proceedings of NAACL-HLT 2018, pages 1551–1563 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension∗ Simon ˇSuster and Walter Daelemans Computational Linguistics & Psycholinguistics Research Center, University of Antwerp, Belgium {simon.suster,walter.daelemans}@uantwerpen.be Abstract We present a new dataset for machine comprehension in the medical domain. Our dataset uses clinical case reports with around 100,000 gap-filling queries about these cases. We apply several baselines and state-of-the-art neural readers to the dataset, and observe a considerable gap in perfor- mance (20% F1) between the best human and machine readers. We analyze the skills required for successful answering and show how reader performance varies depending on the applicable skills. We find that infer- ences using domain knowledge and object tracking are the most frequently required skills, and that recognizing omitted infor- mation and spatio-temporal reasoning are the most difficult for the machines. 1 Introduction Machine comprehension is a task in which a sys- tem reads a text passage and then answers questions about it. The progress in machine comprehension heavily depends on the introduction of new datasets (Burges, 2013), which encourages the development of new algorithms and deepens our understanding of the (linguistic) challenges that can or can not be tackled well by these algorithms. Recently, a number of reading comprehension datasets have been proposed (§ 2), differing in various aspects such as mode of construction, answer-query for- mulation and required understanding skills. Most are open-domain datasets built from news, fiction and Wikipedia texts. For specialized domains, however, large machine comprehension datasets are extremely scarce (Welbl et al., 2017a), and ∗We provide the information about accessing the dataset, as well as the code for the experiments, at http://github. com/clips/clicr. passage: [...] A gradual improvement in clinical and laboratory status was achieved within 20 days of antituberculous treat- ment . The patient was then subjected to a thoracic CT scan that also showed significant radiological improvement . Thereafter , tapering of corticosteroids was initiated with no clinical relapse . The patient was discharged after be- ing treated for a total of 30 days and continued receiving antituberculous therapy with no reported problems for a total of 6 months under the supervision of his hometown physicians . [. . . ] query: If steroids are used , great caution should be exercised on their gradual tapering to avoid . answer: relapse (sem type=problem, cui=C0035020) Figure 1: An example from the dataset, with the pas- sage sentence relevant for answering italicized. The passage has been shortened for clarity. the required comprehension skills poorly under- stood. With our work we hope to narrow this gap by proposing a new resource for reading compre- hension in the clinical domain, and by analyzing the different types of comprehension skills that are triggered while answering (Sugawara et al., 2017; Lai et al., 2017). Machine comprehension for healthcare and medicine has received little attention so far, al- though it offers great potential for practical use. A typical …

Content truncated to 500 words