GGP: Glossary Guided Post-processing for Word Embedding Learning

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4726–4730 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4726 GGP: Glossary Guided Post-processing for Word Embedding Learning Ruosong Yang, Jiannong Cao, Zhiyuan Wen The Hong Kong Polytechnic University Hong Kong, China {csryang,csjcao,cszwen}@comp.polyu.edu.hk Abstract Word embedding learning is the task to map each word into a low-dimensional and continuous vector based on a large corpus. To enhance corpus based word embedding models, researchers utilize domain knowledge to learn more distinguishable representations via joint optimization and post-processing based models. However, joint optimization based models require much training time. Existing post-processing models mostly consider semantic knowledge so that learned embedding models show less functional information. Compared with semantic knowledge sources, glossary is a comprehensive linguistic resource which contains complete semantics. Previous glossary based post-processing method only processed words occurred in the glossary, and did not distinguish multiple senses of each word. In this paper, to make better use of glossary, we utilize attention mechanism to integrate multiple sense representations which are learned respectively. With measuring similarity between word representation and combined sense representation, we aim to capture more topical and functional information. We propose GGP (Glossary Guided Post-processing word embedding) model which consists of a global post-processing function to fine-tune each word vector, and an auto-encoding model to learn sense representations, furthermore, constrains each post-processed word representation and the composition of its sense representations to be similar. We evaluate our model by comparing it with two state-of-the-art models on six word topical/functional similarity datasets, and the re- sults show that it outperforms competitors by an average of 4.1% across all datasets. And our model outperforms GloVe by more than 7%. Keywords: Word Embedding, Post-processing model, Representation Learning 1. Introduction Word embedding learning is the task to utilize a continu- ous vector to represent each word. With the success of Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), which are trained on a large corpus via a sim- ple neural network or matrix factorization, pre-trained word representations are widely used in various Natural Lan- guage Processing (NLP) tasks, such as sequence labeling task (Lample et al., 2016), text classification (Kim, 2014), etc. However, typical word embedding models including Word2Vec and GloVe, are based on the Distributional Hy- pothesis (Harris, 1954), which utilizes the distribution of context words as the target word representation. In prac- tice, the corpus is always limited, which makes it difficult to calculate the actual context word distribution of each tar- get word. For example, synonyms and antonyms are usu- ally hard to distinguish. Meanwhile, the quality of the word embedding highly depends on the frequency of the word in the corpus, rare words are usually discarded. To improve the quality of the word embedding, vari- ous knowledge bases such as WordNet (Miller, 1995), FrameNet (Baker et al., 1998), and Paraphrase Database (Ganitkevitch et al., 2013) are considered. Meanwhile, joint optimization and post-processing are two popu- lar approaches to …

Content truncated to 500 words