Multi-class Hierarchical Question Classification for Multiple Choice Science Exams

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5370–5382 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5370 Multi-class Hierarchical Question Classification for Multiple Choice Science Exams Dongfang Xu∗, Peter Jansen∗, Jaycie Martin†, Zhengnan Xie†, Vikas Yadav∗, Harish Tayyar Madabushi‡, Oyvind Tafjord§ and Peter Clark§ ∗School of Information, University of Arizona, Tucson, AZ, USA †Department of Linguistics, University of Arizona, Tucson, AZ, USA ‡School of Computer Science, University of Birmingham, Birmingham, UK §Allen Insitute for Artificial Intelligence, Seattle, WA, USA pajansen@email.arizona.edu Abstract Prior work has demonstrated that question classification (QC), recognizing the problem domain of a question, can help answer it more accurately. However, developing strong QC algorithms has been hindered by the limited size and complexity of annotated data available. To address this, we present the largest challenge dataset for QC, containing 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains. We then show that a BERT-based model trained on this dataset achieves a large (+0.12 MAP) gain compared with previous methods, while also achieving state-of-the-art performance on benchmark open-domain and biomedical QC datasets. Finally, we show that using this model’s predictions of question topic significantly improves the accuracy of a question answering system by +1.7% P@1, with substantial future gains possible as QC performance improves. Keywords: question answering, question classification 1. Introduction Understanding what a question is asking is one of the first steps that humans use to work towards an answer. In the context of question answering, question classification allows automated systems to intelligently target their inference systems to domain-specific solvers capable of addressing specific kinds of questions and problem solving methods with high confidence and answer accuracy (Hovy et al., 2001; Moldovan et al., 2003). To date, question classification has primarily been studied in the context of open-domain TREC questions (Voorhees and Tice, 2000), with smaller recent datasets available in the biomedical (Roberts et al., 2014; Wasim et al., 2019) and education (Godea and Nielsen, 2018) domains. The open- domain TREC question corpus is a set of 5,952 short factoid questions paired with a taxonomy developed by Li and Roth (2002) that includes 6 coarse answer types (such as entities, locations, and numbers), and 50 fine-grained types (e.g. spe- cific kinds of entities, such as animals or vehicles). While a wide variety of syntactic, semantic, and other features and classification methods have been applied to this task, culminating in near-perfect classification performance (Mad- abushi and Lee, 2016), recent work has demonstrated that QC methods developed on TREC questions generally fail to transfer to datasets with more complex questions such as those in the biomedical domain (Roberts et al., 2014), likely due in part to the simplicity and syntactic regularity of the questions, and the ability for simpler term-frequency models to achieve near-ceiling performance (Xia et al., 2018). In this work we explore question classification in the con- text of multiple choice science exams. Standardized science exams have …

Content truncated to 500 words