GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5866–5871, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5866 GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level Zixian Huang, Yulin Shen, Xiao Li, Yuang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, Yuzhong Qu National Key Laboratory for Novel Software Technology, Nanjing University, China {zixianhuang,ylshen,xiaoli,weiyuang}@smail.nju.edu.cn, {gcheng,daixinyu,yzqu}@nju.edu.cn,zhoul@nlp.nju.edu.cn Abstract Scenario-based question answering (SQA) has attracted increasing research attention. typically requires retrieving and integratingIt knowledge from multiple sources, and apply- ing general knowledge to a specific case de- scribed by a scenario. SQA widely exists in the medical, geography, and legal domains— both in practice and in the exams. In this pa- per, we introduce the GeoSQA dataset. It con- sists of 1,981 scenarios and 4,110 multiple- choice questions in the geography domain at high school level, where diagrams (e.g., maps, charts) have been manually annotated with natural language descriptions to benefit NLP research. Benchmark results on a variety of state-of-the-art methods for question an- swering, textual entailment, and reading com- prehension demonstrate the unique challenges presented by SQA for future research. 1 Introduction Scenario-based question answering (SQA) is an emerging application of NLP (Lally et al., 2017). Different from traditional QA, a question in SQA is accompanied by a scenario, e.g., a patient sum- mary in the medical domain asking for diagnosis or treatment. A scenario differs from a document given in the reading comprehension task where the answer can be extracted or abstracted from the document (Rajpurkar et al., 2016; Nguyen et al., 2016; Lai et al., 2017). SQA requires retrieving and integrating knowledge from multiple sources, and applying general knowledge to a specific case described by the scenario. SQA has found application in many fields, es- pecially in the legal domain (Ye et al., 2018; Luo et al., 2017; Zhong et al., 2018) and in high-school geography exams (Ding et al., 2018; Zhang et al., 2018). The latter is particularly challenging be- cause a geographical scenario consists of both text and diagrams (e.g., maps, charts). Questions in- clude city planning, climates, agriculture plan- ning, transportation, etc. An example of a scenario and a question is presented in Figure 1. Geographical SQA has posed great challenges to NLP and related research, ranging from sce- nario understanding to cross-modal knowledge in- tegration and reasoning. However, there is a lack of large datasets and benchmarking efforts for this task. In this paper, we introduce GeoSQA—an SQA dataset in the geography domain consist- ing of 1,981 scenarios and 4,110 multiple-choice questions at high school level. In particular, each diagram has been manually annotated with a high- quality natural language description of its content, as illustrated in Figure 1. This labor-intensive effort significantly extends the use of GeoSQA, which can support visual SQA (using the dia- grams), natural language based SQA (using the annotations of diagrams), and even the diagram- to-text research. …

Content truncated to 500 words