Learning Word Vectors for Sentiment Analysis

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 142–150, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Learning Word Vectors for Sentiment Analysis Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts Stanford University Stanford, CA 94305 [amaas, rdaly, ptpham, yuze, ang, cgpotts]@stanford.edu Abstract Unsupervised vector-based approaches to se- mantics can model rich lexical meanings, but they largely fail to capture sentiment informa- tion that is central to many word meanings and important for a wide range of NLP tasks. We present a model that uses a mix of unsuper- vised and supervised techniques to learn word vectors capturing semantic term–document in- formation as well as rich sentiment content. The proposed model can leverage both con- tinuous and multi-dimensional sentiment in- formation as well as non-sentiment annota- tions. We instantiate the model to utilize the document-levelsentiment polarity annotations present in many online documents (e.g. star ratings). We evaluate the model using small, widely used sentiment and subjectivity cor- pora and find it out-performs several previ- ously introduced methods for sentiment clas- sification. We also introduce a large dataset of movie reviews to serve as a more robust benchmark for work in this area. 1 Introduction Word representations are a critical component of many natural language processing systems. It is common to represent words as indices in a vocab- ulary, but this fails to capture the rich relational structure of the lexicon. Vector-based models do much better in this regard. They encode continu- ous similarities between words as distance or angle between word vectors in a high-dimensional space. The general approach has proven useful in tasks such as word sense disambiguation, named entity recognition, part of speech tagging, and document retrieval (Turney and Pantel, 2010; Collobert and Weston, 2008; Turian et al., 2010). In this paper, we present a model to capture both semantic and sentiment similarities among words. The semantic component of our model learns word vectors via an unsupervised probabilistic model of documents. However, in keeping with linguistic and cognitive research arguing that expressive content and descriptive semantic content are distinct (Ka- plan, 1999; Jay, 2000; Potts, 2007), we find that this basic model misses crucial sentiment informa- tion. For example, while it learns that wonderful and amazing are semantically close, it doesn’t cap- ture the fact that these are both very strong positive sentiment words, at the opposite end of the spectrum from terrible and awful. Thus, we extend the model with a supervised sentiment component that is capable of embracing many social and attitudinal aspects of meaning (Wil- son et al., 2004; Alm et al., 2005; Andreevskaia and Bergler, 2006; Pang and Lee, 2005; Goldberg and Zhu, 2006; Snyder and Barzilay, 2007). This component of the model uses the vector represen- tation of words to predict the sentiment annotations on contexts in which the words appear. This causes words expressing similar sentiment to have similar vector representations. The full objective function of …

Content truncated to 500 words