Investigating Dynamic Routing in Tree-Structured LSTM for Sentiment Analysis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3432–3437, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3432 Abstract Deep neural network models such as long short-term memory (LSTM) and tree- LSTM have been proven to be effective for sentiment analysis. However, sequential LSTM is a bias model wherein the words in the tail of a sentence are more heavily emphasized than those in the header for building sentence representations. Even tree-LSTM, with useful structural information, could not avoid the bias problem because the root node will be dominant and the nodes in the bottom of the parse tree will be less emphasized even though they may contain salient information. To overcome the bias problem, this study proposes a capsule tree-LSTM model, introducing a dynamic routing algorithm as an aggregation layer to build sentence representation by assigning different weights to nodes according to their contributions to prediction. Experiments on Stanford Sentiment Treebank (SST) for sentiment classification and EmoBank for regression show that the proposed method improved the performance of tree-LSTM and other neural network models. In addition, the deeper the tree structure, the bigger the improvement. 1 Introduction In sentiment analysis, word embeddings (Mikolov et al., 2013a; Mikolov et al., 2013b; Pennington et al., 2014) and sentiment embeddings (Tang et al., 2016; Yu et al., 2018a; Yu et al., 2018b) have become a fundamental component to build deep neural networks such as convolutional neural networks (CNN) (Kalchbrenner et al., 2014; Kim, 2014), recurrent neural networks (RNN) (Graves, 2012; Irsoy and Cardie, 2014), gated recurrent unit (GRU) (Cho et al., 2014), and long short-term memory (LSTM) (Tai et al., 2015; Wang et al., 2015). Given a variable-length text, one challenge of using these neural networks is to compose individual word vectors into sentence vectors with the same length (Iyyer et al., 2015; Joulin et al., 2016; Bojanowski et al., 2016). The sequential neural networks such as RNN, GRU, and LSTM are commonly used due to their ability to capture long-distance dependency in sequential texts. However, these methods belong to the biased model, where the words in the tail of a sentence are more heavily emphasized than those in the header for building sentence representations. As shown in Fig. 1(a), the priority for each word vector will be “fantastic reaall ris rstre this”. This prioritization seems satisfactory for this sentence, but note that the key components could appear anywhere in the sentence rather than necessarily at the end. To improve the abovementioned sequential models, Tai et al. (2015) and Huang et al. (2017) proposed a tree-LSTM model to introduce useful structural information from sentence parse trees. However, the tree-LSTM also heavily emphasizes the root node in the tree to build sentence representations. That is, words that are closed to the root will be given higher priority than words that are far away from the root. As shown in Fig. 1(b), the priority of word …

Content truncated to 500 words