HG2Vec: Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph

Proceedings of the 29th International Conference on Computational Linguistics, pages 3154–3163 October 12–17, 2022. 3154 HG2Vec: Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph Qitong Wang and Mohammed J. Zaki Computer Science Department Rensselaer Polytechnic Institute wangq19@rpi.edu and zaki@cs.rpi.edu Abstract Learning word embeddings is an essential topic in natural language processing. Most existing works use a vast corpus as a primary source while training, but this requires massive time and space for data pre-processing and model training. We propose a new model, HG2Vec, that learns word embeddings utilizing only dic- tionaries and thesauri. Our model reaches the state-of-art on multiple word similarity and re- latedness benchmarks. We demonstrate that dictionaries and thesauri are effective resources to learn word embeddings. In addition, we ex- ploit a new context-focused loss that models transitive relationships between word pairs and balances the performance between similarity and relatedness benchmarks, yielding superior results. 1 Introduction Word embeddings are highly effective for various applications, ranging from recommender systems to named entity recognition (Kubal and Nimkar, 2019). They aim to map words into vectors in a high dimensional space, such that a higher simi- larity between word embeddings captures a closer semantic relationship, whereas a low or negative similarity indicates unrelated or opposite meaning. As a result, predicting whether the embeddings of two words are similar or not is a common approach to evaluate the quality of the embeddings. There are two main perspectives to describe the relationship between a pair of words according to Hill et al. (2015): similarity and relatedness. Simi- larity means that the two words can substitute for each other without generating grammatical mis- takes, while relatedness means that the two words always appear together in a context. For example, coffee and tea is a word pair with high similar- ity and low relatedness. Being popular breakfast drinks, they share many similar properties (both physically and grammatically), but they are not related to each other. On the other hand, green and tea is a word pair with high relatedness and low similarity since green tea is a common type of tea. They do not share similar proprieties, but it is common for them to appear together in a sentence. Most existing works utilize a large corpus to train word embeddings. However, as pointed out by Kiela et al. (2015), models based on associa- tion data, like Wikipedia corpus, have better perfor- mance on relatedness tasks, whereas models based on thesauri have better performance on similarity tasks. Therefore, in this paper, we propose a new model, HG2Vec, that only relies on dictionaries and thesauri to construct contexts via a heteroge- neous graph. We demonstrate that this method can improve the performance of relatedness tasks while maintaining high accuracy on similarity tasks. Our main contributions are as follows: • We utilize dictionaries and thesauri as the only resources (without using any text corpus) to train word embeddings, and obtain state-of-art results on several benchmarks. • We demonstrate that learning synonyms and antonyms is necessary …

Content truncated to 500 words