COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 883–898 August 1–6, 2021. ©2021 Association for Computational Linguistics 883 COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences Shikhar Singh∗1, Nuan Wen∗1, Yu Hou1, Pegah Alipoormolabashi2, Te-Lin Wu3, Xuezhe Ma1, Nanyun Peng3 1University of Southern California, 2 Sharif University of Technology, 3University of California, Los Angeles {ssingh43,nuanwen,houyu,xuezhe.ma}@usc.edu palipoor976@gmail.com {telinwu,violetpeng}@cs.ucla.edu Abstract Commonsense reasoning is intuitive for hu- mans but has been a long-term challenge for artificial intelligence (AI). Recent advance- ments in pretrained language models have shown promising results on several common- sense benchmark datasets. However, the relia- bility and comprehensiveness of these bench- marks towards assessing model’s common- sense reasoning ability remains unclear. To this end, we introduce a new commonsense reasoning benchmark dataset comprising nat- ural language true/false statements, with each sample paired with its complementary coun- terpart, resulting in 4k sentence pairs. We propose a pairwise accuracy metric to reliably measure an agent’s ability to perform common- sense reasoning over a given situation. The dataset is crowdsourced and enhanced with an adversarial model-in-the-loop setup to incen- tivize challenging samples. To facilitate a sys- tematic analysis of commonsense capabilities, we design our dataset along the dimensions of knowledge domains, reasoning scenarios and numeracy. Experimental results demonstrate that our strongest baseline (UnifiedQA-3B), af- ter fine-tuning, achieves ~71% standard accu- racy and ~51% pairwise accuracy, well below human performance (~95% for both metrics). The dataset is available at https://github. com/PlusLabNLP/Com2Sense. 1 Introduction The capability of acquiring and reasoning over com- monsense knowledge plays a crucial role for arti- ficial intelligence (AI) systems that interact with humans and accomplish tasks in the real world. For example, given a situation where someone is asleep, an agent should choose to broom instead of vacuum to clean the room, as the latter would ∗ indicates equal contributions True / False Expecting ten fish in the net, Sammy was thrilled to see forty fish swimming in there. Expecting ten fish in the net, Sammy was thrilled to see five fish swimming in there. S O C I A L C A U S A L N U M E R I C A L P H Y S I C A L C O M P A R A T I V E T / F ? True False As Bob is afraid of heights, he rode the carousel instead of the ferris wheel. As Bob is afraid of heights, he rode the ferris wheel instead of the carousel. T E M P O R A L C O M P A R A T I V E True / False It suddenly snows, so they will spend more time on the road getting home than usual. It suddenly snows, so they will spend a similar amount of time on the road getting home as usual. T / F ? False False False True T / F ? Figure 1: Complementary sentence pair samples from COM2SENSE defined along knowledge domains (e.g. physical), reasoning scenarios (e.g. comparative) …

Content truncated to 500 words