Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1343–1353, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory Xin Wang1, Yuanchao Liu1, Chengjie Sun1, Baoxun Wang2 and Xiaolong Wang1 1School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China 2Application and Service Group, Microsoft, Beijing, China 1{xwang,lyc,cjsun,wangxl}@insun.hit.edu.cn 2baoxwang@microsoft.com Abstract In this paper, we introduce Long Short- Term Memory (LSTM) recurrent network for twitter sentiment prediction. With the help of gates and constant error carousels in the memory block structure, the model could handle interactions between words through a flexible compositional function. Experiments on a public noisy labelled data show that our model outperforms sev- eral feature-engineering approaches, with the result comparable to the current best data-driven technique. According to the evaluation on a generated negation phrase test set, the proposed architecture dou- bles the performance of non-neural model based on bag-of-word features. Further- more, words with special functions (such as negation and transition) are distin- guished and the dissimilarities of words with opposite sentiment are magnified. An interesting case study on negation expres- sion processing shows a promising poten- tial of the architecture dealing with com- plex sentiment phrases. 1 Introduction Twitter and other similar microblogs are rich re- sources for opinions on various kinds of products and events. Detecting sentiment in microblogs is a challenging task that has attracted increasing re- search interest in recent years (Hu et al., 2013b; Volkova et al., 2013). Go et al. (2009) carried out the pioneer work of predicting sentiment in tweets using machine learning technology. They conducted comprehen- sive experiments on multiple classifiers based on bag-of-words feature. Such feature is widely used because it’s simple and surprisingly efficient in many tasks. However, there are also disadvan- tages of bag-of-words features represented by one- hot vectors. Firstly, it bears a data sparsity is- sue (Saif et al., 2012a). In tweets, irregulari- ties and 140-character limitation exacerbate the sparseness. Secondly, losing sequence informa- tion makes it difficult to figure out the polarity properly (Pang et al., 2002). A typical case is that the sentiment word in a negation phrase tends to express opposite sentiment to that of the context. Distributed representations of words can ease the sparseness, but there are limitations to the unsupervised-learned ones. Words with special functions in specific tasks are not distinguished. Such as negation words, which play a special role in polarity classification, are represented sim- ilarly with other adverbs. Such similarities will limit the compositional models’ abilities of de- scribing a sentiment-specific interaction between words. Moreover, word vectors trained by co- occurrence statistics in a small window of con- text effectively represent the syntactic and seman- tic similarity. Thus, words like good and bad have very similar representations (Socher et al., 2011). It’s problematic for sentiment classifiers. Sentiment is expressed by phrases rather than by words (Socher et al., …

Content truncated to 500 words