Manifold Learning-based Word Representation Refinement Incorporating Global and Local Information

Proceedings of the 28th International Conference on Computational Linguistics, pages 3401–3412 Barcelona, Spain (Online), December 8-13, 2020 3401 Manifold Learning-based Word Representation Refinement Incorporating Global and Local Information Wenyu Zhao 1, 3, Dong Zhou1, , Lin Li2, Jinjun Chen3 1. School of Computer Science and Engineering, Hunan University of Science and Technology, Xiangtan, Hunan, 411201, China 2. School of Computer Science and Technology, Wuhan University of Technology, Wuhan, Hubei, 430070, China 3. Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn, Melbourne, VIC, 3122, Australia {wenyuzhao1993, dongzhou1979}@hotmail.com cathylilin@whut.edu.cn jinjun.chen@gmail.com Abstract Recent studies show that word embedding models often underestimate similarities between sim- ilar words and overestimate similarities between distant words. This results in word similarity results obtained from embedding models inconsistent with human judgment. Manifold learning- based methods are widely utilized to refine word representations by re-embedding word vectors from the original embedding space to a new refined semantic space. These methods mainly focus on preserving local geometry information through performing weighted locally linear combination between words and their neighbors twice. However, these reconstruction weights are easily influenced by different selections of neighboring words and the whole combination process is time-consuming. In this paper, we propose two novel word representation refinement methods leveraging isometry feature mapping and local tangent space respectively. Unlike pre- vious methods, our first method corrects pre-trained word embeddings by preserving global geometry information of all words instead of local geometry information between words and their neighbors. Our second method refines word representations by aligning original and re- fined embedding spaces based on local tangent space instead of performing weighted locally linear combination twice. Experimental results obtained from standard semantic relatedness and semantic similarity tasks show that our methods outperform various state-of-the-art baselines for word representation refinement. 1 Introduction Semantic word representations are normally represented as dense, distributed and fixed-length word vectors that are generated by different word embedding models. They can be used to discover some semantic information among words and measure the semantic relatedness of words. Not surprisingly, word vectors and word embedding models have been attracting a lot of attention in the research com-  Corresponding author. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://crea- tivecommons.org/licenses/by/4.0/. 3402 munity. These word embeddings have been proved to be quite useful in a number of Information Re- trieval (IR) tasks, such as machine translation (Mujjiga et al., 2019), text classification (Stein et al.,2019), question answering (Esposito et al., 2020 ) and ad-hoc retrieval (Bagheri et al., 2018; Roy et al. 2018). The performance of aforementioned tasks critically depends on the quality of word embeddings gen- erated from different models. There exist a large number of word embedding models such as BERT (Devlin et al., 2019), C&W (Collobert et al., 2011), Continuous Bag-of-Words (CBOW) (Mikolov et al., 2013a), Skip-Gram (Mikolov et al., 2013b), GloVe (Pennington et al., 2014) and many others (Qiu et al., 2014; Niu et al., 2017; Peng and Zhou, 2020). BERT and its successor models …

Content truncated to 500 words