Rethinking Attribute Representation and Injection for Sentiment Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5602–5613, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5602 Rethinking Attribute Representation and Injection for Sentiment Classification Reinald Kim Amplayo Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh reinald.kim@ed.ac.uk Abstract Text attributes, such as user and product infor- mation in product reviews, have been used to improve the performance of sentiment classi- fication models. The de facto standard method is to incorporate them as additional biases in the attention mechanism, and more perfor- mance gains are achieved by extending the model architecture. In this paper, we show that the above method is the least effective way to represent and inject attributes. To demon- strate this hypothesis, unlike previous models with complicated architectures, we limit our base model to a simple BiLSTM with atten- tion classifier, and instead focus on how and where the attributes should be incorporated in the model. We propose to represent attributes as chunk-wise importance weight matrices and consider four locations in the model (i.e., em- bedding, encoding, attention, classifier) to in- ject attributes. Experiments show that our pro- posed method achieves significant improve- ments over the standard approach and that at- tention mechanism is the worst location to in- ject attributes, contradicting prior work. We also outperform the state-of-the-art despite our use of a simple base model. Finally, we show that these representations transfer well to other tasks1. 1 Introduction The use of categorical attributes (e.g., user, topic, aspects) in the sentiment analysis community (Kim and Hovy, 2004; Pang and Lee, 2007; Liu, 2012) is widespread. Prior to the deep learning era, these information were used as effective cat- egorical features (Li et al., 2011; Tan et al., 2011; Gao et al., 2013; Park et al., 2015) for the ma- chine learning model. Recent work has used them to improve the overall performance (Chen et al., 1 Model implementation and datasets are released here: https://github.com/rktamplayo/CHIM 2016; Dong et al., 2017), interpretability (Am- playo et al., 2018a; Angelidis and Lapata, 2018), and personalization (Ficler and Goldberg, 2017) of neural network models in different tasks such as sentiment classification (Tang et al., 2015), re- view summarization (Yang et al., 2018a), and text generation (Dong et al., 2017). In particular, user and product information have been widely incorporated in sentiment classifica- tion models, especially since they are important metadata attributes found in review websites. Tang et al. (2015) first showed significant accuracy in- crease of neural models when these information are used. Currently, the accepted standard method is to use them as additional biases when comput- ing the weights a in the attention mechanism, as introduced by Chen et al. (2016) as: a = softmax(v⊤e) (1) e = tanh(Wh + Wuu + Wpp + b) (2) = tanh(Wh + bu + bp + b) (3) = tanh(Wh + b′) (4) where u and p are the user and …

Content truncated to 500 words