Do NLP Models Know Numbers? Probing Numeracy in Embeddings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5307–5315, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5307 Do NLP Models Know Numbers? Probing Numeracy in Embeddings Eric Wallace∗ 1, Yizhong Wang∗ 2, Sujian Li2, Sameer Singh3, Matt Gardner1 1Allen Institute for Artificial Intelligence 2Peking University 3University of California, Irvine {ericw, mattg}@allenai.org, {yizhong, lisujian}@pku.edu.cn, sameer@uci.edu Abstract The ability to understand and work with num- bers (numeracy) is critical for many complex reasoning tasks. Currently, most NLP models treat numbers in text in the same way as other tokens—they embed them as distributed vec- tors. Is this enough to capture numeracy? We begin by investigating the numerical reason- ing capabilities of a state-of-the-art question answering model on the DROP dataset. We find this model excels on questions that require numerical reasoning, i.e., it already captures numeracy. To understand how this capabil- ity emerges, we probe token embedding meth- ods (e.g., BERT, GloVe) on synthetic list max- imum, number decoding, and addition tasks. A surprising degree of numeracy is naturally present in standard embeddings. For exam- ple, GloVe and word2vec accurately encode magnitude for numbers up to 1,000. Fur- thermore, character-level embeddings are even more precise—ELMo captures numeracy the best for all pre-trained methods—but BERT, which uses sub-word units, is less exact. 1 Introduction Neural NLP models have become the de-facto standard tool across language understanding tasks, even solving basic reading comprehension and textual entailment datasets (Yu et al., 2018; Devlin et al., 2019). Despite this, existing models are in- capable of complex forms of reasoning, in partic- ular, we focus on the ability to reason numerically. Recent datasets such as DROP (Dua et al., 2019), EQUATE (Ravichander et al., 2019), or Mathe- matics Questions (Saxton et al., 2019) test numer- ical reasoning; they contain examples which re- quire comparing, sorting, and adding numbers in natural language (e.g., Figure 2). The first step in performing numerical reason- ing over natural language is numeracy: the abil- ∗Equal contribution; work done while interning at AI2. −500 0 500 Predicted Value (a) Word2Vec (b) GloVe −500 0 500 Predicted Value (c) ELMo (d) BERT −2000 −1000 0 1000 2000 I put Number −500 0 500 Predicted Value (e) Char-CNN −2000 −1000 0 1000 2000 I put Number (f) Char-LSTM Figure 1: We train a probing model to decode a num- ber from its word embedding over a random 80% of the integers from [-500, 500], e.g., “71” → 71.0. We plot the model’s predictions for all numbers from [-2000, 2000]. The model accurately decodes numbers within the training range (in blue), i.e., pre-trained embed- dings like GloVe and BERT capture numeracy. How- ever, the probe fails to extrapolate to larger numbers (in red). The Char-CNN (e) and Char-LSTM (f) are trained jointly with the probing model. ity to understand and work with numbers in ei- ther digit or word form (Spithourakis and Riedel, 2018). For example, one …

Content truncated to 500 words