Learning Numeral Embedding

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2586–2599 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2586 Learning Numeral Embedding Chengyue Jiang⋄, Zhonglin Nian⋄, Kaihao Guo⋄, Yinggong Zhao†, Shanbo Chu†, Libin Shen†, Kewei Tu⋄∗ ⋄ School of Information Science and Technology, ShanghaiTech University Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Shanghai Engineering Research Center of Intelligent Vision and Imaging † Leyan Technology. Inc {jiangchy, nianzhl, guokh, tukw}@shanghaitech.edu.cn {ygzhao, chushb, libin}@leyantech.com Abstract Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numer- als, a special subset of words, is still under- explored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training cor- pora are highly scarce. In this paper, we pro- pose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self- organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype num- ber embeddings. Numeral embeddings repre- sented in this manner can be plugged into exist- ing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. 1 Introduction Word embeddings have become an essential build- ing block for deep learning approaches to natural language processing (NLP). The quality of pre- trained word embeddings has been shown to sig- nificantly impact the performance of neural ap- proaches to a variety of NLP tasks. Over the past two decades, significant progress has been made in the development of word embedding tech- niques (Lund and Burgess, 1996; Bengio et al., 2003; Bullinaria and Levy, 2007; Mikolov et al., 2013b; Pennington et al., 2014). However, existing word embedding methods do not handle numerals ∗Corresponding author. adequately and cannot directly encode the numer- acy and magnitude of a numeral Naik et al. (2019). Most methods have a limited vocabulary size and therefore can only represent a small subset of the infinite number of numerals. Furthermore, most numerals have very scarce appearances in training corpora and therefore are more likely to be out- of-vocabulary (OOV) compared to non-numerical words. For example, numerals account for 6.15% of all unique tokens in English Wikipedia, but in GloVe (Pennington et al., 2014) which is partially trained on Wikipedia, only 3.79% of its vocabulary is numerals. Previous work (Spithourakis et al., 2016) also shows that the numeral OOV problem is more se- vere when learning word embeddings from corpora with abundant numerals such as clinical reports. Even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the …

Content truncated to 500 words