Learning Embeddings for Rare Words Leveraging Internet Search Engine and Spatial Location Relationships

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 278–287 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 278 Learning Embeddings for Rare Words Leveraging Internet Search Engine and Spatial Location Relationships Xiaotao Li, Shujuan You, Yawen Niu, Wai Chen China Mobile Research Institute, Beijing, China {lixiaotao, youshujuan, niuyawen}@chinamobile.com wai.w.chen@gmail.com Abstract Word embedding techniques depend heavily on the frequencies of words in the corpus, and are negatively impacted by failures in provid- ing reliable representations for low-frequency words or unseen words during training. To ad- dress this problem, we propose an algorithm to learn embeddings for rare words based on an Internet search engine and the spatial loca- tion relationships. Our algorithm proceeds in two steps. We firstly retrieve webpages corre- sponding to the rare word through the search engine and parse the returned results to extract a set of most related words. We average the vectors of the related words as the initial vec- tor of the rare word. Then, the location of the rare word in the vector space is iteratively fine- tuned according to the order of its relevances to the related words. Compared to other ap- proaches, our algorithm can learn more accu- rate representations for a wider range of vocab- ulary. We evaluate our learned rare-word em- beddings on the word relatedness task, and the experimental results show that our algorithm achieves state-of-the-art performance. 1 Introduction Since Bengio et al. (2003) introduced the idea of learning continuous vectors for words using network-based language models, many word em- bedding techniques have been proposed such as Word2vec (Mikolov et al., 2013a,b), GloVe (Pennington et al., 2014), etc. However, nearly all existing word embedding approaches need words that have a high frequency in the corpus and can- not learn good representations for rare words (in- cluding low-frequency words and unseen words). As words in a corpus follow a Zipfian distribution, only a small proportion of the total tokens are fre- quent words, while most of them are rare words. Therefore, how to learn qualified embeddings for rare words is an essential issue to be solved. From the human perspective, when encounter- ing a new word, it is an instinct to take a look at its structure or to look up its definition in a dictionary. The essence of both behaviours is to transform a rare word to a set of familiar words expressing the same meaning to it. Based on the above ideas, some proposed techniques have attempted to ex- ploit subword information or lexical resources to predict the rare word representation. In the area of subword-based approaches, Fast- Text (Bojanowski et al., 2017) learns representa- tions for character n-grams and represents words as the sum of the n-gram vectors. Ngram2vec (Zhao et al., 2017) learns n-gram representations from n-gram co-occurrence statistics and incor- porates this information into the word represen- tations. Pinter et al. (2017) proposed the Mim- ick model to predict vectors for out-of-vocabulary words by learning a function from spellings …

Content truncated to 500 words