Word-Node2Vec: Improving Word Embedding with Document-Level Non-Local Word Co-occurrences

Proceedings of NAACL-HLT 2019, pages 1041–1051 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1041 Word-Node2Vec: Improving Word Embedding with Document-Level Non-Local Word Co-occurrences Procheta Sen1 procheta.sen2@mail.dcu.ie Debasis Ganguly2 debasga1@ie.ibm.com Gareth J.F. Jones1 Gareth.Jones@dcu.ie 1 ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland 2 IBM Research, Dublin, Ireland Abstract Standard word embedding algorithms, such as word2vec and Glove, make a restric- tive assumption that words are likely to be semantically related only if they co-occur lo- cally within a window of fixed size. How- ever, this restrictive assumption may not cap- ture the semantic association between words that co-occur frequently but non-locally within documents. To alleviate this restriction, in this paper, we propose a graph-based word embedding method, named ‘word-node2vec’. By relaxing the strong constraint of local- ity, our method is able to capture both lo- cal and non-local co-occurrences. Word- node2vec constructs a weighted graph, where each node represents a word and the weight of an edge between two nodes represents a com- bination of both local (e.g. document-level co-occurrences.word2vec) and Our experi- ments show that word-node2vec outperforms word2vec and glove on a range of different tasks, such as word-pair similarity prediction, word analogy and concept categorization. 1 Introduction Word embedding, the process of obtaining vec- tor representations of words, is a first step towards addressing language semantics, in which discrete entities, such as words, are embedded as vectors over a continuous space of reals. This not only facilitates to obtain semantic similarities between words to improve tasks such as semantic search (Ganguly et al., 2015; Roy et al., 2016), but is also useful in a number of down-stream NLP tasks in- cluding concept categorization (Jastrzebski et al., 2017), information retrieval (Guo et al., 2016), sentence similarity prediction (Mueller and Thya- garajan, 2016), sentiment analysis (Faruqui et al., 2015) and POS tagging (Tsvetkov et al., 2016) etc. Word embedding approaches such as word2vec (Mikolov et al., 2013a) and Glove (Pennington et al., 2014) rely on a large corpus to learn the association between words. The archi- tecture of existing word embedding approaches mimics the process of human cognition of word association by learning the representation of each word with an objective of maximizing the likelihood of predicting the words around its local context (defined by a fixed length word window). A limitation of existing word embedding ap- proaches, such as word2vec and glove, is that they use a strong constraint that words are likely to be semantically related to each other only if one occurs within a local context of the another, where the local context is given by a word window of specified length. On the other hand, non-local or document-level co-occurrences between words have been widely used to estimate semantic similarities between words. More specifically, the latent semantic anal- ysis (LSA) method proposed by Deerwester et al. (1990) uses a spectral analysis (method of prin- cipal component analysis) of the term-document matrix of a collection to obtain the most informa- …

Content truncated to 500 words