Do Language Embeddings capture Scales?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4889–4896 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4889 Do Language Embeddings capture Scales? Xikun Zhang∗† Stanford University xikunz2@cs.stanford.edu Deepak Ramachandran∗ Google Research ramachandrand@google.com Ian Tenney Google Research iftenney@google.com Yanai Elazar Bar Ilan Univesity, AI2 yanaiela@gmail.com Dan Roth University of Pennsylvania danroth@seas.upenn.edu Abstract Pretrained Language Models (LMs) have been shown to possess significant linguistic, com- mon sense and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pre- trained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key fac- tors affecting their performance, and show that a simple method of canonicalizing numbers can have a significant effect on the results. 1 1 Introduction The success of contextualized pretrained Language Models like BERT (Devlin et al., 2018) and ELMo (Peters et al., 2018) on tasks like Question Answer- ing and Natural Language Inference, has led to speculation that they are good at Common Sense Reasoning (CSR). On one hand, recent work has approached this question by measuring the ability of LMs to an- swer questions about physical common sense (Bisk et al., 2020) (”How to separate egg whites from yolks?”), temporal reasoning (Zhou et al., 2020) (”How long does a basketball game take?”), and numerical common sense (Lin et al., 2020). On the other hand, after realizing some high-level reason- ing skills like this may be difficult to learn from a language-modeling objective only, (Geva et al., 2020) injects numerical reasoning skills into LMs by additional pretraining on automatically gener- ated data. All of these skills are prerequisites for CSR. ∗Both authors contributed equally. † Work done during an internship at Google Research. 1Code and models are available at https://github. com/google-research-datasets/numbert. Figure 1: Scalar probing example. The mass of ”dog” is a distribution (gray histogram) concentrated around 10-100kg. We train a linear model over a frozen (shown by the snowflake in the figure) encoder to predict this distribution (orange histogram) using either a dense cross-entropy or a regression loss (Section 3). Here, we address a simpler task which is another pre-requisite for CSR: the prediction of scalar at- tributes, a task we call Scalar Probing. Given an object (such as a ”wedding ring”) and an attribute with continuous numeric values (such as Mass or Price), can an LM’s representation of the object predict the value of that attribute? Since in gen- eral, there may not be a single correct value for such attributes due to polysemy (“crane” as a bird, versus construction equipment) or natural variation (e.g. different breeds of dogs), we interpret this as a task of predicting a distribution of possible values for this attribute, and compare it to a ground truth distribution of such values. An overview of this scalar probing is shown in Figure 1. …

Content truncated to 500 words