Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8154 - 8163 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings Jiangbin Zheng†, Yile Wang†, Ge Wang, Jun Xia, Yufei Huang, Guojiang Zhao, Yue Zhang, Stan Z. Li* School of Engineering, Westlake University Institute of Advanced Technology, Westlake Institute for Advanced Study {zhengjiangbin,wangyile,wangge,xiajun,huangyufei, zhaoguojiang,zhangyue,Stan.ZQ.Li}@westlake.edu.cn Abstract Although contextualized embeddings gener- ated from large-scale pre-trained models per- form well in many tasks, traditional static em- beddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low compu- tational cost, ease of deployment, and stabil- ity. In this paper, we aim to improve word em- beddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings inde- pendent of training by employing priori syn- onym knowledge and weighted vector distri- bution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin. 1 Introduction Contextualized embeddings such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) have become the default architectures for most down- stream NLP tasks. However, they are computation- ally expensive, resource-demanding, hence envi- ronmentally unfriendly. Compared with contex- tualized embeddings, static embeddings like Skip- gram (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are lighter and less computationally expensive. Furthermore, they can even perform without significant performance loss for context- independent tasks like lexical-semantic tasks (e.g., word analogy), or some tasks with plentiful labeled data and simple language (Arora et al., 2020). Recent work has attempted to enhance static word embedding while maintaining the benefits of both contextualized embedding and static em- bedding. Among these efforts, one category is the direct conversion of contextualized embeddings † Co-first author. * Corresponding author. Trained Word Embeddings Graph-based Retrofitting Refined Word Embeddings Static Model Contextualized Model Monolingual Corpus Post-processing Context-to-Vec Figure 1: The overall training pipeline of our proposed word embeddings training and post-processing methods. In the Context-to-Vec phase, static word embeddings are trained using contextualized embeddings based on a monolingual corpus. While in the post-processing phase, external knowledge is introduced to fine-tune the word vectors based on the graph topology. to static embeddings (Bommasani et al., 2020). The other category of enhancement is to make use of contextualized embeddings for static embed- dings (Melamud et al., 2016). The latter category is a newer paradigm, which we call Context-to-Vec. This paradigm not only alleviates the word sense ambiguities from static embedding, but also fuses more syntactic and semantic information in the context within a fixed window. For the Context-to-Vec paradigm, an association between contextualized word vectors and static word vectors is essentially required. In this case, the contextualized signal serves as a source of information enhancement for the static embed- dings (Vashishth et al., 2018). However, the ex- isting efforts only …

Content truncated to 500 words