Methods for Numeracy-Preserving Word Embeddings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4742–4753, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4742 Methods for Numeracy-Preserving Word Embeddings Dhanasekar Sundararaman1, Shijing Si1, Vivek Subramanian1, Guoyin Wang2, Devamanyu Hazarika3, Lawrence Carin1 1 Duke University 2 Amazon Alexa AI 3 National University of Singapore dhanasekar.sundararaman@duke.edu Abstract Word embedding models are typically able to capture the semantics of words via the distri- butional hypothesis, but fail to capture the nu- merical properties of numbers that appear in a text. This leads to problems with numerical reasoning involving tasks such as question an- swering. We propose a new methodology to assign and learn embeddings for numbers. Our approach creates Deterministic, Independent- of-Corpus Embeddings (referred to as DICE) for numbers, such that their cosine similarity reflects the actual distance on the number line. DICE outperforms a wide range of pre-trained word embedding models across multiple exam- ples of two tasks: (i) evaluating the ability to capture numeration and magnitude; and (ii) to perform list maximum, decoding, and addition. We further explore the utility of these embed- dings in downstream applications by initializ- ing numbers with our approach for the task of magnitude prediction. We also introduce a reg- ularization approach to learn model-based em- beddings of numbers in a contextual setting. 1 Introduction Word embeddings capture semantic relationships between words by operationalizing the distribu- tional hypothesis (Harris, 1954; Firth, 1957). They can be learned either non-contextually (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) or contextually (Devlin et al., 2018; Peters et al., 2018). Non-contextual embeddings have worked well on various language understand- ing and semantic tasks (Rumelhart et al., 1988; Mikolov et al., 2013a,b). More recently, they have also been used as pre-trained word embeddings to aid more sophisticated contextual models for solv- ing rigorous natural language processing (NLP) problems, including translation, paraphrasing, and sentence-similarity tasks (Kiros et al., 2015; Wiet- ing et al., 2015). While word embeddings effectively capture se- mantic relationships between words, they are less effective at capturing numeric properties associated with numbers. Though numbers represent a signif- icant percentage of tokens in a corpus, they are often overlooked. In non-contextual word embed- ding models, they are treated like any other word, which leads to misinterpretation. For instance, they exhibit unintuitive similarities with other words and do not contain strong prior information about the magnitude of the number they encode. In sentence similarity and reasoning tasks, failure to handle numbers causes as much as 29% of contradictions (De Marneffe et al., 2008). In other data-intensive tasks where numbers are abundant, like neural ma- chine translation, they are masked to hide the trans- lation models inefficiency in dealing with them (Mitchell and Lapata, 2009). There are a variety of tests proposed to measure the efficiency of number embeddings. For instance, Naik et al. (2019) shows that GloVe (Pennington et al., 2014), word2vec (Mikolov et al., 2013b), and fastText (Joulin et al., 2016; Bojanowski et al., 2017) fail to …

Content truncated to 500 words