SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3695–3706 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3695 SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics Da Yin♣, Tao Meng♠, Kai-Wei Chang♠ ♣ Peking University ♠ University of California, Los Angeles wade yin9712@pku.edu.cn, tmeng@cs.ucla.edu, kw@kwchang.net Abstract We propose SentiBERT, a variant of BERT that effectively captures compositional sen- timent semantics. The model incorporates contextualized representation with binary con- stituency parse tree to capture semantic com- position. Comprehensive experiments demon- strate that SentiBERT achieves competi- tive performance on phrase-level sentiment classification. We further demonstrate that the sentiment composition learned from the phrase-level annotations on SST can be trans- ferred to other sentiment analysis tasks as well as related tasks, such as emotion clas- sification tasks. Moreover, we conduct ab- lation studies and design visualization meth- ods to understand SentiBERT. We show that SentiBERT is better than baseline ap- proaches in capturing negation and the con- trastive relation and model the compositional sentiment semantics. 1 Introduction Sentiment analysis is an important language pro- cessing task (Pang et al., 2002, 2008; Liu, 2012). One of the key challenges in sentiment analysis is to model compositional sentiment semantics. Take the sentence “Frenetic but not really funny.” in Fig- ure 1 as an example. The two parts of the sentence are connected by “but”, which reveals the change of sentiment. Besides, the word “not” changes the sentiment of “really funny”. These types of nega- tion and contrast are often difficult to handle when the sentences are complex (Socher et al., 2013; Tay et al., 2018; Xu et al., 2019). In general, the sentiment of an expression is de- termined by the meaning of tokens and phrases and the way how they are syntactically combined. Prior studies consider explicitly modeling compositional sentiment semantics over constituency structure with recursive neural networks (Socher et al., 2012, ������ ����� � ��� ��� �������� �������� ������� �������� Figure 1: Illustration of the challenges of learning sen- timent semantic compositionality. The blue nodes rep- resent token nodes. The colors of phrase nodes in the binary constituency tree represent the sentiment of phrases. The red boxes show that the sentiment changes from the child node to the parent node due to negation and contrast. 2013). However, these models that generate repre- sentation of a parent node by aggregating the local information from child nodes, overlook the rich association in context. In this paper, we propose SentiBERT to in- corporate recently developed contextualized rep- resentation models (Devlin et al., 2019; Liu et al., 2019) with the recursive constituency tree structure to better capture compositional sentiment seman- tics. Specifically, we build a simple yet effective attention network for composing sentiment seman- tics on top of BERT (Devlin et al., 2019). During training, we follow BERT to capture contextual information by masked language modeling. In ad- dition, we instruct the model to learn composition of meaning by predicting sentiment labels of the phrase nodes. Results on phrase-level …

Content truncated to 500 words