Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6862–6868, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6862 Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models Bill Yuchen Lin Seyeon Lee Rahul Khanna Xiang Ren {yuchen.lin,seyeonle,rahulkha,xiangren}@usc.edu Department of Computer Science, University of Southern California Abstract Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as “neural knowledge bases” via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical com- monsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a di- agnostic dataset, NUMERSENSE1, containing 13.6k masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our anal- ysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on the diag- nostic dataset prior to any fine-tuning; (2) fine- tuning with distant supervision brings some improvement; (3) the best supervised model still performs poorly as compared to human performance (54.06% vs. 96.3% in accuracy). 1 Introduction Pre-trained language models (PTLMs), such as BERT (Devlin et al., 2019), have yielded state- of-the-art performance on many natural language processing tasks. Given PTLMs’ cited ability to create general, yet useful text representations, an investigation of their ability to encode com- monsense knowledge into representations is war- ranted––commonsense knowledge is often required to have a full understanding of language. Recently there have been a few recent works that do investigate the inquiry of whether PTLMs possess commonsense knowledge (Petroni et al., 2019; Davison et al., 2019; Bouraoui et al., 2020). Overall, these prior studies suggest that PTLMs are 1https://inklab.usc.edu/NumerSense/ 1st:fly (79.5%) 2nd:sing (9.1%) 1st:four(44.8%) 2nd:two (18.7%) Birds can [MASK]. A bird usually has [MASK] legs. A car usually has [MASK] wheels. A car usually has [MASK] round wheels. However, for Numerical Commonsense Knowledge : 1st:four(53.7%) 2nd:two (20.5%) 1st:two (37.1%) 2nd:four(20.2%) BERT-Large Masked Word Prediction Figure 1: Top: PTLMs often cannot solve masked language modeling tasks needing numerical commonsense knowledge, hence our title. Bottom: Even when PTLMs seemingly suc- ceed, they fail to stay consistent under small perturbations. creating text representations that often have com- monsense knowledge encoded in them. We, how- ever, find it surprising that when posed with a simi- lar reasoning-based masked-word-prediction task, PTLMs perform poorly in recalling the required numerical commonsense knowledge (see Figure 1). Therefore, in this paper, our goal is to study whether PTLMs capture numerical commonsense knowledge, i.e., commonsense knowledge that pro- vides an understanding of the numeric relation be- tween entities. We propose measuring this capa- bility via a masked-word-prediction based probing task, where, the ranking of numeric words by what the model believes most probably fills the mask would expose the capabilities of PTLMs to capture numeric commonsense knowledge. …

Content truncated to 500 words