Automatic Question Answering for Medical MCQs: Can It go Further than Information Retrieval?

Proceedings of Recent Advances in Natural Language Processing, pages 418–422, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_049 418 Automatic Question Answering for Medical MCQs: Can It Go Further than Information Retrieval? Le An Ha and Victoria Yaneva Research Institute in Information and Language Processing, University of Wolverhampton, UK {ha.l.a, v.yaneva}@wlv.ac.uk Abstract We present a novel approach to automatic question answering that does not depend on the performance of an information retrieval (IR) system and does not require training data. We evaluate the system performance on a chal- lenging set of university-level medical science multiple-choice questions. Best performance is achieved when combining a neural approach with an IR approach, both of which work independently. Unlike previous approaches, the system achieves statistically significant im- provement over the random guess baseline even for questions that are labeled as chal- lenging based on the performance of baseline solvers. 1 Introduction Automatic question answering has seen a renewed interest in recent years as a challenge problem for evaluating machine intelligence. This has driven the development of large-scale question- answering data sets such as SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), WikiMovies benchmark (Chen et al., 2017), Triv- iaQA (Joshi et al., 2017) (to name a few), as well as the organisation of workshops such as the Machine Reading for Question Answering 2018 workshop1. In spite of the optimistic ad- vances over crowd-sourced questions and online queries, automatic question answering for real exam questions is still a very challenging and under-explored area. For example, the Allen AI Science Challenge2 invited researchers worldwide to develop systems that could solve standardized eight-grade science questions. The best system out of all 780 participating teams achieved a score of 59.31% correct answers using a combination of 1https://mrqa2018.github.io/ 2https://www.kaggle.com/c/the-allen-ai-science- challenge 15 gradient-boosting models (random baseline of 25%), while the authors report that using Informa- tion Retrieval (IR) alone results in a score of 55%. The difficulties related to answering exam ques- tions are partly due to the complexity of the rea- soning involved and partly to the lack of large training data. Another significant reason is the fact that the existing approaches to question answer- ing are dependent on the performance of IR sys- tems and can rarely go far beyond the performance of such systems. While IR is a powerful method when answering questions where the correct an- swer is a string contained within a document, the systems fail when the sentences within the ques- tion do not individually hold a clue to what the correct answer might be (Clark et al., 2018). This is one of the characteristics of Multiple Choice Questions (MCQs) from the science domain that makes them so challenging for both machines and for humans. In this paper we aim to address these short- comings by developing an approach that: i) does not require that the training data (often unavail- able) be in the form of multiple-choice ques- tions and ii) does not depend on matching strings of text with one another. We use …

Content truncated to 500 words