ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1122–1136, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1122 ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning Michael Boratko∗ Xiang Lorraine Li∗ Tim O’Gorman∗ Rajarshi Das∗ Dan Le Andrew McCallum College of Information and Computer Sciences University of Massachusetts Amherst {mboratko,xiangl,rajarshi,togorman,dhle,mccallum}@cs.umass.edu Abstract Given questions regarding some prototypical situation — such as Name something that peo- ple usually do before they leave the house for work? — a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answer- ing dataset for training and evaluating com- mon sense reasoning capabilities of artificial intelligence systems in such prototypical sit- uations. The training set is gathered from an existing set of questions played in a long- running international game show – FAMILY- FEUD. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. After pre- senting multiple competitive baseline models, we find that human performance still exceeds model scores on all evaluation metrics with a meaningful gap, supporting the challenging na- ture of the task. 1 Introduction Humans possess the ability to implicitly reason us- ing a wealth of common background knowledge, much of which is acquired through shared experi- ences. For example, consider the question in Fig- ure 1 — “Name something that people usually do before they leave the house for work.”. Humans can agree about the details and characteristics of a pro- totypical event or situation (Schank and Abelson, 1975, 1977) due to commonalities in their shared lived experiences, cultural norms and expectations. This rough agreement extends beyond an agree- ment on a single top response, but can be viewed ∗ Equal contribution. (i) Name something that people usually do before they leave for work? Ask 100 crowd-workers + manual clustering Groom Lock door Shower Breakfast Dress Say goodbye Pray Jog Invalid Answers 5 7 43 30 7 4 1 1 2 (ii) Name a piece of equipment that you are likely to find at your office and not at home? Categories: printer/copier (37), office furniture (15), computer equipment (17), stapler (11) files (10), office appliances (5), security systems (1) (iii) Name something that an athlete would not keep in her refrigerator. Categories: unhealthy / fast food (36), unhealthy drinks (24), clothing/shoes (24) accessories (7) (iv) Name something that you might forget in a hotel room? Categories: phone (24), toothbrush/towels (17), clothing/shoes (15) keys (14), purse/wallet (14), accessories (8), charger (5) Figure 1: We focus on common-sense reasoning over prototypical situations when there could be many dif- ferent answers but some are more common than others. Our task is in generative style (not multiple-choice for- mat). Answers to a question are crowd-sourced …

Content truncated to 500 words