Improved Word Embeddings with Implicit Structure Information

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2408–2417, Osaka, Japan, December 11-17 2016. Improved Word Embeddings with Implicit Structure Information Jie Shen Cong Liu∗ School of Data and Computer Science, Sun Yat-sen University shenjie5@mail2.sysu.edu.cn, liucong3@mail.sysu.edu.cn Abstract Distributed word representation is an efficient method for capturing semantic and syntactic word relations. In this work, we introduce an extension to the continuous bag-of-words model for learning word representations efficiently by using implicit structure information. Instead of rely- ing on a syntactic parser which might be noisy and slow to build, we compute weights represent- ing probabilities of syntactic relations based on the Huffman softmax tree in an efficient heuristic. The constructed “implicit graphs” from these weights show that these weights contain useful im- plicit structure information. Extensive experiments performed on several word similarity and word analogy tasks show gains compared to the basic continuous bag-of-words model. 1 Introduction Unsupervised word embeddings have been shown to improve many downstream NLP tasks, such as dependency parsing (Chen and Manning, 2014; Kong et al., 2014), part-of-speech tagging (Collobert et al., 2011), sentiment analysis (Socher et al., 2013) and machine translation (Devlin et al., 2014). Low- dimensional embeddings are generally learnt in a language model by maximizing the likelihood of a large corpus of raw text data. Word vectors that are close to each other are semantically related based on the distributional hypothesis (Harris, 1954), which states that words in similar contexts have similar meanings. Based on the distributional hypothesis, many methods of building word vectors were explored. Exam- ples of these are SENNA (Collobert and Weston, 2008), the hierarchical log-bilinear model (Mnih and Hinton, 2009), Word2Vec (Mikolov et al., 2013a; Mikolov et al., 2013b) and GloVe (Pennington et al., 2014). In this work, we introduce an extension to the continuous bag-of-words (CBOW) model (Mikolov et al., 2013b). The CBOW model is widely used for learning word embeddings from raw textual data, popularized via the Word2Vec tool. Not only does it build useful word embeddings, but it is also efficient for training and scales well to huge corpora. In (Levy and Goldberg, 2014), based on the fact that nearby words are not necessarily syntactically related, word context is derived from dependency parse-trees relying on a syntactic parser instead of simply using the surrounding words. Only the words that have dependency relations with the center word are used as the context words, as illustrated in Figure 1. However, syntactic parsing is a more difficult and time-consuming task than finding word embeddings. The challenge is to absorb the advantage of using syntactic information while avoiding the complexity of parsing. In this paper, we use a simple method to attach different weights to the context words to approximate the context obtained from explicit syntactic trees, such as dependency parse-trees. The method of obtaining contextual weights is based on the Huffman softmax tree in softmax period. Our method is as efficient as CBOW since we do not explicitly construct syntactic trees but only …

Content truncated to 500 words