Importance of Self-Attention for Sentiment Analysis

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 267–275 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 267 Importance of Self-Attention for Sentiment Analysis Ga¨el Letarte∗, Fr´ed´erik Paradis∗, Philippe Gigu`ere, Franc¸ois Laviolette Department of Computer Science and Software Engineering Universit´e Laval, Qu´ebec, Canada {gael.letarte, frederik.paradis}.1@ulaval.ca Abstract Despite their superior performance, deep learning models often lack interpretability. In this paper, we explore the modeling of insight- ful relations between words, in order to un- derstand and enhance predictions. To this ef- fect, we propose the Self-Attention Network (SANet), a flexible and interpretable archi- tecture for text classification. Experiments indicate that gains obtained by self-attention is task-dependent. For instance, experiments on sentiment analysis tasks showed an im- provement of around 2% when using self- attention compared to a baseline without atten- tion, while topic classification showed no gain. Interpretability brought forward by our archi- tecture highlighted the importance of neigh- boring word interactions to extract sentiment. 1 Introduction Deep neural networks have achieved great suc- cesses on numerous tasks. However, they are of- ten seen as black boxes, lacking interpretability. Research efforts in order to solve this issue have steadily increased (Simonyan et al., 2013; Zeiler and Fergus, 2014; Bach et al., 2015; Ribeiro et al., 2016; Fong and Vedaldi, 2017). In language mod- eling, interpretability often takes place via an at- tention mechanism in the neural network (Bah- danau et al., 2014; Xu et al., 2015; Sukhbaatar et al., 2015; Choi et al., 2017). In this context, attention essentially allows a network to identify which words in a sentence are more relevant. Be- yond interpretability, this often results in improved decision making by the network. Recently, Vaswani et al. (2017) proposed the Transformer architecture for machine translation. It relies only on attention mechanisms, instead of making use of either recurrent or convolutional ∗Authors contributed equally to this work. neural networks. This architecture contains lay- ers called self-attention (or intra-attention) which allow each word in the sequence to pay attention to other words in the sequence, independently of their positions. We modified this architecture, re- sulting in the following contributions: • A novel architecture for text classification called Self-Attention Network (SANet) that models the interactions between all input word pairs. It is sequence length-agnostic, thanks to a global max pooling layer. • A study on the impact of this self-attention mechanism on large scale datasets. In partic- ular, we empirically demonstrate the positive impact of self-attention in terms of perfor- mance and interpretability for sentiment anal- ysis, compared to topic classification. In the study, we make use of two quantitative met- rics (Gini coefficient and diagonality) that ex- hibit particular behaviors for attention mech- anisms in sentiment analysis. 2 Related Work The majority of text classification techniques ei- ther use convolutional or recurrent neural net- works on the words or the characters of the sen- tence (Zhang et al., 2015, 2017; Yang et al., 2016; Conneau et al., 2017; Johnson and Zhang, 2016, …

Content truncated to 500 words