Two/Too Simple Adaptations of Word2Vec for Syntax Problems

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1299–1304, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Two/Too Simple Adaptations of Word2Vec for Syntax Problems Wang Ling Chris Dyer Alan Black Isabel Trancoso L2F Spoken Systems Lab, INESC-ID, Lisbon, Portugal Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA Instituto Superior T´ecnico, Lisbon, Portugal {lingwang,cdyer,awb}@cs.cmu.edu isabel.trancoso@inesc-id.pt Abstract We present two simple modifications to the models in the popular Word2Vec tool, in or- der to generate embeddings more suited to tasks involving syntax. The main issue with the original models is the fact that they are insensitive to word order. While order in- dependence is useful for inducing semantic representations, this leads to suboptimal re- sults when they are used to solve syntax-based problems. We show improvements in part-of- speech tagging and dependency parsing using our proposed models. 1 Introduction Word representations learned from neural language models have been shown to improve many NLP tasks, such as part-of-speech tagging (Collobert et al., 2011), dependency parsing (Chen and Man- ning, 2014; Kong et al., 2014) and machine trans- lation (Liu et al., 2014; Kalchbrenner and Blunsom, 2013; Devlin et al., 2014; Sutskever et al., 2014). These low-dimensional representations are learned as parameters in a language model and trained to maximize the likelihood of a large corpus of raw text. They are then incorporated as features along side hand-engineered features (Turian et al., 2010), or used to initialize the parameters of neural net- works targeting tasks for which substantially less training data is available (Hinton and Salakhutdinov, 2012; Erhan et al., 2010; Guo et al., 2014). One of the most widely used tools for building word vectors are the models described in (Mikolov et al., 2013), implemented in the Word2Vec tool, in particular the “skip-gram” and the “continuous bag-of-words” (CBOW) models. These two mod- els make different independence and conditioning assumptions; however, both models discard word order information in how they account for context. Thus, embeddings built using these models have been shown to capture semantic information be- tween words, and pre-training using these models has been shown to lead to major improvements in many tasks (Collobert et al., 2011). While more so- phisticated approaches have been proposed (Dhillon et al., 2011; Huang et al., 2012; Faruqui and Dyer, 2014; Levy and Goldberg, 2014; Yang and Eisen- stein, 2015), Word2Vec remains a popular choice due to their efficiency and simplicity. However, as these models are insensitive to word order, embeddings built using these models are sub- optimal for tasks involving syntax, such as part-of- speech tagging or dependency parsing. This is be- cause syntax defines “what words go where?”, while semantics than “what words go together”. Obvi- ously, in a model where word order is discarded, the many syntactic relations between words can- not be captured properly. For instance, while most words occur with the word the, only nouns tend to occur exactly afterwords (e.g. the cat). This is supported …

Content truncated to 500 words