Contextual Bidirectional Long Short-Term Memory Recurrent Neural Network Language Models: A Generative Approach to Sentiment Analysis

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1023–1032, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Contextual Bidirectional Long Short-Term Memory Recurrent Neural Network Language Models: A Generative Approach to Sentiment Analysis Amr El-Desoky Mousa1 and Bj¨orn Schuller1,2 1Chair of Complex & Intelligent Systems, University of Passau, Passau, Germany 2Department of Computing, Imperial College London, London, UK amr.mousa@uni-passau.de schuller@ieee.org Abstract Traditional learning-based approaches to sentiment analysis of written text use the concept of bag-of-words or bag-of-n- grams, where a document is viewed as a set of terms or short combinations of terms disregarding grammar rules or word order. Novel approaches de-emphasize this concept and view the problem as a sequence classification problem. In this context, recurrent neural networks (RNNs) have achieved significant success. The idea is to use RNNs as discriminative bi- nary classifiers to predict a positive or neg- ative sentiment label at every word posi- tion then perform a type of pooling to get a sentence-level polarity. Here, we investi- gate a novel generative approach in which a separate probability distribution is esti- mated for every sentiment using language models (LMs) based on long short-term memory (LSTM) RNNs. We introduce a novel type of LM using a modified version of bidirectional LSTM (BLSTM) called contextual BLSTM (cBLSTM), where the probability of a word is estimated based on its full left and right contexts. Our ap- proach is compared with a BLSTM binary classifier. Significant improvements are observed in classifying the IMDB movie review dataset. Further improvements are achieved via model combination. 1 Introduction Sentiment analysis of text (also known as opinion mining) is the process of computationally identify- ing and categorizing opinions expressed in a piece of text in order to determine the writer’s attitude towards a particular topic. Due to the tremendous increase in web content, many organizations be- came increasingly interested in analyzing this big data in order to monitor the public opinion and as- sist decision making. Therefore, sentiment analy- sis attracted the interest of many researchers. The task of sentiment analysis can be seen as a text classification problem. Depending on the tar- get of the analysis, the classes can be described by continuous primitives such as valence, a polarity state (positive or negative attitude), or a subjectiv- ity state (objective or subjective). In this work, we are interested in the binary classification of doc- uments into a positive or negative attitude. Such detection of polarity is a non-trivial problem due to the existence of noise, comparisons, vocabulary changes, and the use of idioms, irony, and domain specific terminology (Schuller et al., 2015). Traditional approaches to sentiment analysis rely on the concept of bag-of-words or bag-of-n- grams, where a document is viewed as a set of terms or short combinations of terms disregarding grammar rules or word order. In this case, usually, the analysis involves: tokenization and parsing of text documents, careful selection of important fea- tures (terms), dimensionality …

Content truncated to 500 words