HEAD-QA: A Healthcare Dataset for Complex Reasoning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 960 HEAD-QA: A Healthcare Dataset for Complex Reasoning David Vilares Universidade da Coru˜na, CITIC Departamento de Computaci´on Campus de Elvi˜na s/n, 15071 A Coru˜na, Spain david.vilares@udc.es Carlos G´omez-Rodr´ıguez Universidade da Coru˜na, CITIC Departamento de Computaci´on Campus de Elvi˜na s/n, 15071 A Coru˜na, Spain carlos.gomez@udc.es Abstract We present HEAD-QA, a multi-choice ques- tion answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are chal- lenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with in- formation retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its useful- ness as a benchmark for future work. 1 Introduction Recent progress in question answering (QA) has been led by neural models (Seo et al., 2016; Kundu and Ng, 2018), due to their ability to process raw texts. However, some authors (Kaushik and Lip- ton, 2018; Clark et al., 2018) have discussed the tendency of research to develop datasets and meth- ods that accomodate the data-intensiveness and strengths of current neural methods. This is the case of popular English datasets such as bAbI (Weston et al., 2015) or SQuAD (Ra- jpurkar et al., 2016, 2018), where some systems achieve near human-level performance (Hu et al., 2018; Xiong et al., 2017) and often surface-level knowledge suffices to answer. To counteract this, Clark et al. (2016) and Clark et al. (2018) have encouraged progress by developing multi-choice datasets that require reasoning. The questions match grade-school science, due to the difficul- ties to collect specialized questions. With a similar aim, Lai et al. (2017) released 100k questions and 28k passages intended for middle or high school Chinese students, and Zellers et al. (2018) intro- duced a dataset for common sense reasoning from a spectrum of daily situations. Question (medicine): A 13-year-old girl is operated on due to Hirschsprung illness at 3 months of age. Which of the following tumors is more likely to be present? 1. Abdominal neuroblastoma 2. Wilms tumor 3. Mesoblastic nephroma 4. Familial thyroid medullary carcinoma. Question (pharmacology) The antibiotic treatment of choice for Meningitis caused by Haemophilus influenzae serogroup b is: 1. Gentamicin 2. Erythromycin 3. Ciprofloxacin 4. Cefotaxime Question (psychology) According to research derived from the Eysenck model, there is evidence that extraverts, in comparison with introverts: 1. Perform better in surveillance tasks. 2. Have greater salivary secretion before the lemon juice test. 3. Have a greater need for stimulation. 4. Have less tolerance to pain. Table 1: Samples from HEAD-QA However, this kind of dataset is scarce for com- plex domains like medicine: while challenges have been proposed in such domains, like textual entailment (Abacha et al., 2015; Abacha and Dina, 2016) or answering questions about specific doc- uments and …

Content truncated to 500 words