SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6975–6988, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6975 SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge Pei Ke∗, Haozhe Ji∗, Siyang Liu, Xiaoyan Zhu, Minlie Huang† Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China kepei1106@outlook.com,{jhz20,siyang-l18}@mails.tsinghua.edu.cn {zxy-dcs,aihuang}@tsinghua.edu.cn Abstract Most of the existing pre-trained language rep- resentation models neglect to consider the lin- guistic knowledge of texts, which can pro- mote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language repre- sentation model called SentiLARE, which in- troduces word-level linguistic knowledge in- cluding part-of-speech tag and sentiment po- larity (inferred from SentiWordNet) into pre- trained models. We first propose a context- aware sentiment attention mechanism to ac- quire the sentiment polarity of each word with its part-of-speech tag by querying SentiWord- Net. Then, we devise a new pre-training task called label-aware masked language model to construct knowledge-aware language represen- tation. Experiments show that SentiLARE ob- tains new state-of-the-art performance on a va- riety of sentiment analysis tasks1. 1 Introduction Recently, pre-trained language representation mod- els such as GPT (Radford et al., 2018, 2019), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2019) have achieved promising results in NLP tasks, including sentiment analysis (Xu et al., 2019, 2020; Yin et al., 2020). These models cap- ture contextual information from large-scale cor- pora via well-designed pre-training tasks. The literature has commonly reported that pre-trained models can be used as effective feature extractors and achieve state-of-the-art performance on various downstream tasks (Wang et al., 2019a). Despite the great success of pre-trained models, existing pre-training tasks like masked language model and next sentence prediction (Devlin et al., ∗ Equal contribution † Corresponding author 1The data, codes, and model parameters are available at https://github.com/thu-coai/SentiLARE. 2019) neglect to consider the linguistic knowledge. Such knowledge is important for some NLP tasks, particularly for sentiment analysis. For instance, existing work has shown that linguistic knowledge including part-of-speech tag (Qian et al., 2015; Huang et al., 2017) and word-level sentiment po- larity (Qian et al., 2017) is closely related to the sentiment of longer texts. We argue that pre-trained models enriched with the linguistic knowledge of words will facilitate the understanding of the senti- ment of the whole texts, thereby resulting in better performance on sentiment analysis. There are two major challenges to construct knowledge-aware pre-trained language represen- tation models which can promote the downstream tasks in sentiment analysis: 1) Knowledge acqui- sition across different contexts. Most of the ex- isting work has adopted static sentiment lexicons as linguistic resource (Qian et al., 2017; Chen et al., 2019), and equipped each word with a fixed senti- ment polarity across different contexts. However, the same word may play different sentiment roles in different contexts due to the variety of part-of- speech tags …

Content truncated to 500 words