Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4984 - 4994 December 7-11, 2022 ©2022 Association for Computational Linguistics Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis Shuai Fan1, Chen Lin1∗, Haonan Li2†, Zhenghao Lin1†, Jinsong Su1 Hang Zhang3, Yeyun Gong4, Jian Guo3, Nan Duan4 1 School of Informatics, Xiamen University, China 2 The University of Melbourne, Australia 3 IDEA Research, China 4 Microsoft Research Asia Abstract Most existing pre-trained language representa- tion models (PLMs) are sub-optimal in senti- ment analysis tasks, as they capture the senti- ment information from word-level while under- considering sentence-level information. In this paper, we propose SentiWSP, a novel Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks. The word level pre-training task detects replaced sentiment words, via a generator-discriminator framework, to enhance the PLM’s knowledge about sentiment words. The sentence level pre-training task further strengthens the discriminator via a contrastive learning framework, with similar sentences as negative samples, to encode sentiments in a sen- tence. Extensive experimental results show that SentiWSP achieves new state-of-the-art perfor- mance on various sentence-level and aspect- level sentiment classification benchmarks. We have made our code and model publicly avail- able at https://github.com/XMUDM/SentiWSP. 1 Introduction Sentiment analysis plays a fundamental role in natural language processing (NLP) and powers a broad spectrum of important business applica- tions such as marketing (HaCohen-Kerner, 2019) and campaign monitoring (Sandoval-Almazán and Valle-Cruz, 2020). Two typical sentiment analy- sis tasks are sentence-level sentiment classifica- tion (Xu et al., 2019; Yin et al., 2020; Tang et al., 2022) and aspect-level sentiment classification (Li et al., 2021b). Recently, pre-trained language representation models (PLMs) such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018, 2019), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and XL- Net (Yang et al., 2019) have brought impressive per- formance improvements in many NLP problems, *Corresponding author, chenlin@xmu.edu.cn †Equal contribution including sentiment analysis. PLMs learn a robust encoder on a large unlabeled corpus, through care- fully designed pre-training tasks, such as masked token or next sentence prediction. Despite their progress, the application of general purposed PLMs in sentiment analysis is limited, because they fail to distinguish the importance of different words to a specific task. For example, it is shown in (Kassner and Schütze, 2020) that general purposed PLMs have difficulties dealing with con- tradictory sentiment words or negation expressions, which are critical in sentiment analysis. To ad- dress this problem, recent sentiment-aware PLMs introduce word-level sentiment information, such as token sentiments and emoticons (Zhou et al., 2020), aspect word (Tian et al., 2020), word-level linguistic knowledge (Ke et al., 2020), and im- plicit sentiment-knowledge information (Li et al., 2021b). These word-level pre-training tasks, e.g., sentiment word prediction and word polarity predic- tion, mainly learn from the masked words and are not efficient to capture word-level information for all input words. Furthermore, sentiment expressed in a sentence is beyond the simple aggregation of word-level sentiments. However, general purposed PLMs and existing sentiment-aware …

Content truncated to 500 words