CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Proceedings of NAACL-HLT 2019, pages 4149–4158 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4149 COMMONSENSEQA: A Question Answering Challenge Targeting Commonsense Knowledge Alon Talmor∗,1,2 Jonathan Herzig∗,1 Nicholas Lourie2 Jonathan Berant1,2 1School of Computer Science, Tel-Aviv University 2Allen Institute for Artificial Intelligence {alontalmor@mail,jonathan.herzig@cs,joberant@cs}.tau.ac.il, nicholasl@allenai.org Abstract When answering a question, people often draw upon their rich world knowledge in addi- tion to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present COMMONSENSEQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from CON- CEPTNET (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discrim- inate in turn between each of the target con- cepts. This encourages workers to create ques- tions with complex semantics that often re- quire prior knowledge. We create 12,247 ques- tions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and ob- tains 56% accuracy, well below human perfor- mance, which is 89%. 1 Introduction When humans answer questions, they capitalize on their common sense and background knowl- edge about spatial relations, causes and effects, scientific facts and social conventions. For in- stance, given the question “Where was Simon when he heard the lawn mower?”, one can infer that the lawn mower is close to Simon, and that it is probably outdoors and situated at street level. This type of knowledge seems trivial for humans, but is still out of the reach of current natural lan- guage understanding (NLU) systems. ∗ The authors contributed equally river waterfall bridge valley a) Sample ConceptNet for specific subgraphs b) Crowd source corresponding natural language questions and two additional distractors Where on a river can you hold a cup upright to catch water on a sunny day? ✔‍ waterfall, ✘ bridge, ✘ valley, ✘ pebble, ✘ mountain Where can I stand on a river to see water falling without getting wet? ✘ waterfall, ✔ bridge, ✘ valley, ✘ stream, ✘ bottom AtLocation I’m crossing the river, my feet are wet but my body is dry, where am I? ✘ waterfall, ✘ bridge, ✔ valley, ✘ bank, ✘ island pebble stream bank AtLocation AtLocation canyon AtLocation AtLocation AtLocation AtLocation … Figure 1: (a) A source concept (‘river’) and three tar- get concepts (dashed) are sampled from CONCEPT- NET (b) Crowd-workers generate three questions, each having one of the target concepts for its answer (), while the other two targets are not (). Then, for each question, workers choose an additional distractor from CONCEPTNET (in italics), and author one themselves (in bold). Work on Question Answering (QA) has mostly focused on answering factoid questions, where the answer …

Content truncated to 500 words