Investigating Different Syntactic Context Types and Context Representations for Learning Word Embeddings

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2421–2431 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Investigating Different Syntactic Context Types and Context Representations for Learning Word Embeddings Bofang Li1,2 libofang@ruc.edu.cn Tao Liu1,2 tliu@ruc.edu.cn Zhe Zhao1,2 helloworld@ruc.edu.cn Buzhou Tang3 tangbuzhou@gmail.com Aleksandr Drozd4 alex@smg.is.titech.ac.jp Anna Rogers5 arogers@cs.uml.edu Xiaoyong Du1,2 duyong@ruc.edu.cn 1 School of Information, Renmin University of China 2 Key Laboratory of Data Engineering and Knowledge Engineering, MOE 3 Shenzhen Graduate School, Harbin Institute of Technology 4 Global Scientific Information and Computing Center, Tokyo Institute of Technology 5 Department of Computer Science, University of Massachusetts Lowell Abstract The number of word embedding models is growing every year. Most of them are based on the co-occurrence information of words and their contexts. However, it is still an open question what is the best def- inition of context. We provide a system- atical investigation of 4 different syntactic context types and context representation- s for learning word embeddings. Com- prehensive experiments are conducted to evaluate their effectiveness on 6 extrinsic and intrinsic tasks. We hope that this pa- per, along with the published code, would be helpful for choosing the best contex- t type and representation for a given task. 1 Introduction Recently, there is a growing interest in word em- bedding models, where words are embedded into low-dimensional (dense) real-valued vectors. The trained word embeddings can be directly used for solving intrinsic tasks like word similarity and word analogy. They are also helpful for solv- ing extrinsic tasks, such as part-of-speech tagging, chunking, named entity recognition (Collobert and Weston, 2008; Collobert et al., 2011) and text clas- sification (Socher et al., 2013; Kim, 2014). The training objectives of word embedding models are based on the Distributional Hypoth- esis (Harris, 1954) that can be stated as follows: “words that occur in similar contexts tend to have similar meanings”. In most word embedding mod- els, the “context” is defined as the words which precede and follow the target word within some fixed distance (Bengio et al., 2003; Mnih and Hin- ton, 2007; Mikolov et al., 2013a; Pennington et al., 2014). Among them, Global Vectors (GloVe) proposed by Pennington et al. (2014), Contin- uous Skip-Gram (CSG) 1 and Continuous Bag- Of-Words (CBOW) proposed by Mikolov et al. (2013b) achieve state-of-the-art results on a range of linguistic tasks, and scale to corpora with bil- lions of words. The traditional sparse vector-space models have explored many different types of context. Cur- ran (2004); Pad´o and Lapata (2007); Clark (2012) have discussed a set of context definitions beyond simple linear context. For example, a sentence or document could be used as the boundary instead of window size. Contextual words could be associat- ed with their relative sides (left/right) or position- s (+1/-2) to the target word. They could also be associated with part-of-speech or grammatical re- lation labels. The weight of each contextual word can be explicitly defined. Moreover, words that are connected to target word in dependency parse 1Many …

Content truncated to 500 words