Dependency Based Embeddings for Sentence Classification Tasks

Proceedings of NAACL-HLT 2016, pages 1490–1500, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Dependency Based Embeddings for Sentence Classification Tasks Alexandros Komninos Department of Computer Science University of York York, YO10 5GH United Kingdom ak1153@york.ac.uk Suresh Manandhar Department of Computer Science University of York York, YO10 5GH United Kingdom suresh@cs.york.ac.uk Abstract We compare different word embeddings from a standard window based skipgram model, a skipgram model trained using dependency context features and a novel skipgram variant that utilizes additional information from dependency graphs. We explore the effectiveness of the different types of word embeddings for word similarity and sentence classification tasks. We consider three common sentence classification tasks: question type classification on the TREC dataset, binary sentiment classification on Stanford’s Sentiment Treebank and semantic relation classification on the SemEval 2010 dataset. For each task we use three different classification methods: a Support Vector Machine, a Convolutional Neural Network and a Long Short Term Memory Network. Our experiments show that dependency based embeddings outperform standard window based embeddings in most of the settings, while using dependency context embeddings as additional features improves performance in all tasks regardless of the classification method. Our embeddings and code are available at https://www.cs.york.ac.uk/nlp/ extvec 1 Introduction Representing words as low dimensional vectors (also known as word embeddings) has been a widely adopted technique in NLP. Word representations can be used as features for classification tasks such as named entity recognition or chunking (Turian et al., 2010), and as a pretraining method for initializing deep neural network representations (Collobert et al., 2011; Kim, 2014). Word embeddings provide better generalization to unseen examples since they can capture general semantic and syntactic proper- ties of words. One of the most popular methods of learning word embeddings is the skipgram model of Mikolov et al. (2013a; 2013b) where embeddings are trained by making predictions of context words appearing in a window around a target word. The standard skipgram model ignores syntax and only partially takes into consideration the sequen- tial structure of text, but still captures certain syn- tactic properties of words. A significant amount of previous research has explored methods for di- rectly taking syntax into account for word embed- ding learning (Pham et al., 2015; Cheng and Kart- saklis, 2015; Hashimoto et al., 2014). One simple method is based on traditional count-based distribu- tional semantic spaces and utilizes words with syn- tactic types from a dependency parse graph as con- text features (Pad´o and Lapata, 2007; Baroni and Lenci, 2010). This method has also been applied to skipgram models, where word embeddings are op- timized to predict dependency context features in- stead of other words (Levy and Goldberg, 2014). Syntax-based embeddings have been shown to have different properties in word similarity evalua- tions than their window based counterparts, better capturing the functional properties of words. How- ever, it is not clear if they provide any advantage for NLP tasks. We show that using dependency context features can be a general method of providing …

Content truncated to 500 words