Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1660–1669, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification Jiacheng Xu† Danlu Chen‡ Xipeng Qiu∗‡ Xuanjing Huang‡ Software School, Fudan University † School of Computer Science, Fudan University ‡ Shanghai Key Laboratory of Intelligent Information Processing, Fudan University†‡ 825 Zhangheng Road, Shanghai, China†‡ {jcxu13,dlchen13,xpqiu,xjhuang}@fudan.edu.cn Abstract Recently, neural networks have achieved great success on sentiment classification due to their ability to alleviate feature engineering. How- ever, one of the remaining challenges is to model long texts in document-level sentiment classification under a recurrent architecture because of the deficiency of the memory unit. To address this problem, we present a Cached Long Short-Term Memory neural networks (CLSTM) to capture the overall semantic in- formation in long texts. CLSTM introduces a cache mechanism, which divides memory into several groups with different forgetting rates and thus enables the network to keep sentiment information better within a recur- rent unit. The proposed CLSTM outperforms the state-of-the-art models on three publicly available document-level sentiment analysis datasets. 1 Introduction Sentiment classification is one of the most widely used natural language processing techniques in many areas, such as E-commerce websites, online social networks, political orientation analyses (Wil- son et al., 2009; O’Connor et al., 2010), etc. Recently, deep learning approaches (Socher et al., 2013; Kim, 2014; Chen et al., 2015; Liu et al., 2016) have gained encouraging results on sentiment clas- sification, which frees researchers from handcrafted feature engineering. Among these methods, Recur- rent Neural Networks (RNNs) are one of the most ∗ Corresponding author. prevalent architectures because of the ability to han- dle variable-length texts. Sentence- or paragraph-level sentiment analysis expects the model to extract features from limited source of information, while document-level senti- ment analysis demands more on selecting and stor- ing global sentiment message from long texts with noises and redundant local pattern. Simple RNNs are not powerful enough to handle the overflow and to pick up key sentiment messages from relatively far time-steps . Efforts have been made to solve such a scalabil- ity problem on long texts by extracting semantic in- formation hierarchically (Tang et al., 2015a; Tai et al., 2015), which first obtain sentence representa- tions and then combine them to generate high-level document embeddings. However, some of these so- lutions either rely on explicit a priori structural as- sumptions or discard the order information within a sentence, which are vulnerable to sudden change or twists in texts especially a long-range one (Mc- Donald et al., 2007; Mikolov et al., 2013). Re- current models match people’s intuition of reading word by word and are capable to model the intrinsic relations between sentences. By keeping the word order, RNNs could extract the sentence representa- tion implicitly and meanwhile analyze the semantic meaning of a whole document without any explicit boundary. Partially inspired by neural structure of human brain and computer system architecture, we present the Cached Long Short-Term …

Content truncated to 500 words