Document Modeling with Gated Recurrent Neural Network for Sentiment Classification

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Document Modeling with Gated Recurrent Neural Network for Sentiment Classification Duyu Tang, Bing Qin∗, Ting Liu Harbin Institute of Technology, Harbin, China {dytang, qinb, tliu}@ir.hit.edu.cn Abstract Document level sentiment classification remains a challenge: encoding the intrin- sic relations between sentences in the se- mantic meaning of a document. To ad- dress this, we introduce a neural network model to learn vector-based document rep- resentation in a unified, bottom-up fash- ion. The model first learns sentence rep- resentation with convolutional neural net- work or long short-term memory. After- wards, semantics of sentences and their relations are adaptively encoded in docu- ment representation with gated recurren- t neural network. We conduct documen- t level sentiment classification on four large-scale review datasets from IMDB and Yelp Dataset Challenge. Experimen- tal results show that: (1) our neural mod- el shows superior performances over sev- eral state-of-the-art algorithms; (2) gat- ed recurrent neural network dramatically outperforms standard recurrent neural net- work in document modeling for sentiment classification.1 1 Introduction Document level sentiment classification is a fun- damental task in sentiment analysis, and is cru- cial to understand user generated content in so- cial networks or product reviews (Manning and Sch¨utze, 1999; Jurafsky and Martin, 2000; Pang and Lee, 2008; Liu, 2012). The task calls for iden- tifying the overall sentiment polarity (e.g. thumbs up or thumbs down, 1-5 stars on review sites) of a document. In literature, dominant approaches fol- low (Pang et al., 2002) and exploit machine learn- ∗Corresponding author. 1 Codes and datasets are publicly available at http://ir.hit.edu.cn/˜dytang. ing algorithm to build sentiment classifier. Many of them focus on designing hand-crafted features (Qu et al., 2010; Paltoglou and Thelwall, 2010) or learning discriminate features from data, since the performance of a machine learner is heavily de- pendent on the choice of data representation (Ben- gio et al., 2015). Document level sentiment classification re- mains a significant challenge: how to encode the intrinsic (semantic or syntactic) relations between sentences in the semantic meaning of documen- t. This is crucial for sentiment classification be- cause relations like “contrast” and “cause” have great influences on determining the meaning and the overall polarity of a document. However, ex- isting studies typically fail to effectively capture such information. For example, Pang et al. (2002) and Wang and Manning (2012) represent docu- ments with bag-of-ngrams features and build SVM classifier upon that. Although such feature-driven SVM is an extremely strong performer and hardly to be transcended, its “sparse” and “discrete” char- acteristics make it clumsy in taking into account of side information like relations between sentences. Recently, Le and Mikolov (2014) exploit neural networks to learn continuous document represen- tation from data. Essentially, they use local ngram information and do not capture semantic relations between sentences. Furthermore, a person asked to do this task will naturally carry it out in a se- quential, bottom-up fashion, …

Content truncated to 500 words