RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7560–7579 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7560 RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms Pei Zhou Rahul Khanna Seyeon Lee Bill Yuchen Lin Daniel Ho Jay Pujara Xiang Ren Department of Computer Science and Information Sciences Institute University of Southern California {peiz,rahulkha,seyeonle,yuchen.lin,hsiaotuh,jpujara,xiangren}@usc.edu Abstract Pre-trained language models (PTLMs) have achieved impressive performance on common- sense inference benchmarks, but their ability to employ commonsense to make robust infer- ences, which is crucial for effective communi- cations with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference using Commonsense Axioms, that evaluates robust commonsense inference de- spite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowl- edge bases and probe PTLMs across two dif- ferent evaluation settings. Extensive experi- ments on our generated probe sets with more than 10k statements show that PTLMs per- form no better than random guessing on the zero-shot setting, are heavily impacted by sta- tistical biases, and are not robust to pertur- bation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen in- ferences. Our new large-scale benchmark ex- poses a significant gap between PTLMs and human-level language understanding and of- fers a new challenge for PTLMs to demon- strate commonsense.1 1 Introduction Smooth and effective communication requires the ability to make various forms of commonsense in- ferences (Clark and Brennan, 1991). When a friend texts, “I’m going to perform in front of thousands tomorrow,” you may reply reassuringly, “Deep breaths, you’ll do great!” Implicit to this com- munication is a commonsense logical inference that a person performing in front of a crowd may feel anxious, and that a reassuring remark helps ease anxiety (Figure 1). A growing body of liter- ature (Bosselut et al., 2019; Petroni et al., 2019) 1Our code and data are public at https://sites. google.com/usc.edu/rica. Text Message: “I’m going to perform in front of thousands tomorrow…” ---------------------------- Explicit Knowledge: Friend is going to perform in front of many people tomorrow ----------------------------- Commonsense Axiom: Performing in front of people can cause anxiety Text Message “Deep breaths, you’ll do great!” ------------------------------ Inference Made: My friend might be anxious, let me try to calm them Linguistically-Varied Statements of the same Commonsense Axiom • A person performing in front of people might be nervous • People performing in front of people find it harder to be relaxed • It can be hard for someone to be calm when they’re about to perform Figure 1: Human communication requires common- sense inferences. RICA evaluates such inferences via commonsense axioms with many linguistic variations. shows pre-trained language models (PTLMs) are able to catalog the types of commonsense relation- ships necessary for fluid communication. However, as we show in this paper, PTLMs have a shocking inability to leverage such commonsense knowledge to make robust …

Content truncated to 500 words