Specializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs

Proceedings of the 28th International Conference on Computational Linguistics, pages 3599–3609 Barcelona, Spain (Online), December 8-13, 2020 3599 Specializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs Yuanhang Ren1 and Ye Du2 1University of Electronic Science and Technology of China 2Southwestern University of Finance and Economics, China {ryuanhang,henry.duye}@gmail.com Abstract Traditional word vectors, such as word2vec and glove, have a well-known inclination to conflate the semantic similarity with other semantic relations. A retrofitting procedure may be needed to solve this issue. In this work, we propose a new retrofitting method called Heterogeneously Retrofitted Spectral Word Embedding. It heterogeneously twists the similarity matrix of word pairs with lexical constraints. A new set of word vectors is generated by a spectral decomposition of the similarity matrix, which has a linear algebraic analytic form. Our method has a compet- itive performance compared with the state-of-the-art retrofitting method such as AR (Mrkˇsi´c et al., 2017). In addition, since our embedding has a clear linear algebraic relationship with the similarity matrix, we carefully study the contribution of each component in our model. Last but not least, our method is very efficient to execute1. 1 Introduction Word embedding is one of the core research areas in natural language processing. Its usefulness has been demonstrated in a wide variety of NLP tasks, e.g. dependency parsing, sentiment analysis, and machine reading comprehension. Modern embedding methods are usually based on the distributional hypothesis, namely, co-occurred words tend to purport similar semantic meanings. Although the distributional word vectors perform well on lots of tasks, they have a well-known tendency to conflate the semantic similarity information with the semantic relatedness (Hill et al., 2015). Therefore, the similarity between word vectors cannot reflect the precise semantic relation between word pairs, but just a semantic association (Yih et al., 2012). For instance, if two words are antonyms, their corresponding word vectors could be very close geometrically, which makes it very hard to distinguish one word from the other. One way to solve this problem is to inject some lexical constraints, such as antonym relationships into the word vectors, the aim is to make antonyms far apart from each other. This process is often referred as semantic specialization (Mrkˇsi´c et al., 2017). There are two kinds of semantic specialization methods: (1) joint specialization methods, in which word vectors are trained from scratch by incorporating the lexical knowledge into the learning objective of the distributional models. Pham et al. (2015) inject a synonym/antonym margin loss into the skip-gram objective to enforce that antonym pairs have low simi- larity while synonym pairs have high similarity; (2) retrofitting methods (also referred as post-processing methods), in which pre-trained word vectors are fine-tuned by injecting the lexical information into vec- tor spaces. Mrkˇsi´c et al. (2017) proposed the ATTRACT-REPEL(AR) algorithm which tries to push or pull a pair of words by a margin compared with its corresponding negative samples. Another retrofitting method is to inject lexical constraints into the word similarity matrix and obtain the tuned word vectors via …

Content truncated to 500 words