MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8862–8874 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8862 MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset Jing Li, Shangping Zhong and Kaizhi Chen College of Computer and Data Science, Fuzhou University, Fuzhou, China {N190320027, spzhong, ckz}@fzu.edu.cn Abstract Question Answering (QA) has been success- fully applied in scenarios of human-computer interaction such as chatbots and search en- gines. However, for the specific biomedi- cal domain, QA systems are still immature due to expert-annotated datasets being lim- ited by category and scale. In this paper, we present MLEC-QA, the largest-scale Chi- nese multi-choice biomedical QA dataset, col- lected from the National Medical Licensing Examination in China. The dataset is com- posed of five subsets with 136,236 biomedi- cal multi-choice questions with extra materi- als (images or tables) annotated by human ex- perts, and first covers the following biomed- ical sub-fields: Clinic, Stomatology, Public Health, Traditional Chinese Medicine, and Traditional Chinese Medicine Combined with Western Medicine. We implement eight rep- resentative control methods and open-domain QA methods as baselines. Experimental re- sults demonstrate that even the current best model can only achieve accuracies between 40% to 55% on five subsets, especially per- forming poorly on questions that require so- phisticated reasoning ability. We hope the re- lease of the MLEC-QA dataset can serve as a valuable resource for research and evaluation in open-domain QA, and also make advances for biomedical QA systems.1 1 Introduction As a branch of the QA task, Biomedical Question Answering (BQA) enables effectively perceiving, accessing, and understanding complex biomedi- cal knowledge by innovative applications, which makes BQA an important QA application in the biomedical domain (Jin et al., 2021). Such a task has recently attracted considerable attention from the NLP community (Zweigenbaum, 2003; He et al., 2020b; Jin et al., 2020), but is still con- fronted with the following three key challenges: 1https://github.com/Judenpech/MLEC-QA (1) Most work attempt to build BQA systems with deep learning and neural network techniques (Ben Abacha et al., 2017, 2019b; Pampari et al., 2018) and are thus data-hungry. However, annotat- ing large-scale biomedical question-answer pairs with high quality is prohibitively expensive. As a result, current expert-annotated BQA datasets are small in size. (2) Multi-choice QA is a typ- ical format type of BQA dataset. Most previous work focus on such format type of datasets in which contents are in the field of clinical medicine (Zhang et al., 2018b; Jin et al., 2020) and con- sumer health (Zhang et al., 2017, 2018a; He et al., 2019; Tian et al., 2019). However, there are many other specialized sub-fields in biomedicine that have not been studied before (e.g., Stomatology). (3) Ideal BQA systems should not only focus on raw text data, but also fully utilize various types of biomedical resources, such as images and ta- bles. Unfortunately, most BQA datasets are ei- ther texts (Tsatsaronis et al., 2015; Pampari et al., 2018; Jin et al., 2019) or images (Lau et al., 2018; Ben …

Content truncated to 500 words