Social IQa: Commonsense Reasoning about Social Interactions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4463–4473, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4463 SOCIAL IQA: Commonsense Reasoning about Social Interactions Maarten Sap⋆ ♦♥ Hannah Rashkin⋆ ♦♥ Derek Chen♥ Ronan Le Bras♦ Yejin Choi♦♥ ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA ♥Paul G. Allen School of Computer Science & Engineering, Seattle, WA, USA {msap,hrashkin,dchen14,yejin}@cs.washington.edu {ronanlb}@allenai.org Abstract We introduce SOCIAL IQA, the first large- scale benchmark for commonsense reasoning about social situations. SOCIAL IQA contains 38,000 multiple choice questions for prob- ing emotional and social intelligence in a va- riety of everyday situations (e.g., Q: “Jor- dan wanted to tell Tracy a secret, so Jor- dan leaned towards Tracy. Why did Jordan do this?” A: “Make sure no one else could hear”). Through crowdsourcing, we collect commonsense questions along with correct and incorrect answers about social interac- tions, using a new framework that mitigates stylistic artifacts in incorrect answers by ask- ing workers to provide the right answer to a different but related question. Empirical re- sults show that our benchmark is challenging for existing question-answering models based on pretrained language models, compared to human performance (>20% gap). Notably, we further establish SOCIAL IQA as a re- source for transfer learning of commonsense knowledge, achieving state-of-the-art perfor- mance on multiple commonsense reasoning tasks (Winograd Schemas, COPA). 1 Introduction Social and emotional intelligence enables humans to reason about the mental states of others and their likely actions (Ganaie and Mudasir, 2015). For example, when someone spills food all over the floor, we can infer that they will likely want to clean up the mess, rather than taste the food off the floor or run around in the mess (Figure 1, middle). This example illustrates how Theory of Mind, i.e., the ability to reason about the implied emotions and behavior of others, enables humans to nav- igate social situations ranging from simple con- versations with friends to complex negotiations in courtrooms (Apperly, 2010). ⋆ Both authors contributed equally. REASONING ABOUT WHAT HAPPENS NEXT (a) taste the food (b) mop up ✔ (c) run around in the mess What will Alex want to do next? A Alex spilled the food she just prepared all over the floor and it made a huge mess. Q REASONING ABOUT EMOTIONAL REACTIONS (a) sorry for the villain (b) hopeful that Robin will succeed ✔ (c) like Robin should lose How would others feel afterwards? In the school play, Robin played a hero in the struggle to the death with the angry villain. A Q REASONING ABOUT MOTIVATION Why did Tracy do this? Tracy had accidentally pressed upon Austin in the small elevator and it was awkward. (a) get very close to Austin (b) squeeze into the elevator ✔ (c) get flirty with Austin A Q Figure 1: Three context-question-answers triples from SOCIAL IQA, along with the type of reasoning required to answer them. In the …

Content truncated to 500 words