Exploring Numeracy in Word Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374–3380 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3374 Exploring Numeracy in Word Embeddings Aakanksha Naik∗, Abhilasha Ravichander∗, Carolyn Rose, Eduard Hovy Language Technologies Institute, Carnegie Mellon University {anaik, aravicha, cprose, ehovy}@cs.cmu.edu Abstract Word embeddings are now pervasive across NLP subfields as the de-facto method of form- ing text representataions. In this work, we show that existing embedding models are in- adequate at constructing representations that capture salient aspects of mathematical mean- ing for numbers, which is important for lan- guage understanding. Numbers are ubiquitous and frequently appear in text. Inspired by cog- nitive studies on how humans perceive num- bers, we develop an analysis framework to test how well word embeddings capture two es- sential properties of numbers: magnitude (e.g. 3<4) and numeration (e.g. 3=three). Our ex- periments reveal that most models capture an approximate notion of magnitude, but are in- adequate at capturing numeration. We hope that our observations provide a starting point for the development of methods which better capture numeracy in NLP systems. 1 Introduction Word embeddings operationalize the distributional hypothesis, where a word is characterized by “the company it keeps” (Harris, 1954; Firth, 1957), and have been shown to capture semantic regularities in vector space (Mikolov et al., 2013c). They have been a driving force in NLP in recent years, and enjoy widespread use in a variety of semantic tasks (Rumelhart et al.; Mikolov et al., 2013a,b; Col- lobert and Weston, 2008; Glorot et al., 2011; Tur- ney and Pantel, 2010; Turney, 2013). However, to what extent do these word repre- sentations capture numeric properties? Numbers often need to be dealt with precisely, and under- standing the meaning of text also requires a care- ful understanding of the quantities involved. They have been identified to play an important role in textual entailment, a benchmark natural language ∗*The first two authors contributed equally to this work. understanding task. Marneffe et al. (2008) ex- tract pairs of contradictions that occur naturally on Wikipedia and Google News, and find that as many as 29% of contradictions arise due to numeric discrepancies. They also identify that on many Recognizing Textual Entailment (RTE) datasets, 8.8% of contradictory pairs feature nu- meric contradictions. Naik et al. (2018) find that model inability to do numerical reasoning causes 4% of errors made by state-of-the-art models in Natural Language Inference. Spithourakis and Riedel (2018) emphasize the importance of nu- meracy in language modeling. Yet, numbers are often forgotten and even masked in NLP applica- tions (Mitchell and Lapata, 2009). In several domains such as economics, finance and scientific articles numbers can play a crucial role in text. Take for example a recent news head- line, Met Office: Global Warming could exceed 1.5 C within five years Ideally, the text representation we use should be able to capture that global warming can exceed 1.5 C, not 100 C. Magnitude is an essential aspect of a number’s …

Content truncated to 500 words