SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4067–4076 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4067 SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis Hao Tian‡,†, Can Gao†, Xinyan Xiao†, Hao Liu†, Bolei He†, Hua Wu†, Haifeng Wang†, Feng Wu‡ †Baidu Inc., Beijing, China ‡University of Science and Technology of China {tianhao,gaocan01,xiaoxinyan,liuhao24,hebolei,wu hua, wanghaifeng}@baidu.com, fengwu@ustc.edu.cn Abstract Recently, sentiment analysis has seen remark- able advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowl- edge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and con- structs three sentiment knowledge prediction objectives, so as to embed sentiment informa- tion at the word, polarity and aspect level into pre-trained sentiment representation. In partic- ular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aim- ing to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta. 1 Introduction Sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments. In practice, sentiment analysis involves a wide range of specific tasks (Liu, 2012), such as sentence-level sentiment classification, aspect-level sentiment classification, opinion extraction and so on. Traditional meth- ods often study these tasks separately and design specific models for each task, based on manually- designed features (Liu, 2012) or deep learning (Zhang et al., 2018). Recently, pre-training methods (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019) have shown their powerfulness in learning general semantic representations, and have remarkably improved most natural language processing (NLP) tasks like sentiment analysis. These methods build unsupervised objectives at word-level, such as masking strategy (Devlin et al., 2019), next-word prediction (Radford et al., 2018) or permutation (Yang et al., 2019). Such word- prediction-based objectives have shown great abili- ties to capture dependency between words and syn- tactic structures (Jawahar et al., 2019). However, as the sentiment information of a text is seldom ex- plicitly studied, it is hard to expect such pre-trained general representations to deliver optimal results for sentiment analysis (Tang et al., 2014). Sentiment analysis differs from other NLP tasks in that it deals mainly with user reviews other than news texts. There are many specific sentiment tasks, and these tasks usually depend on differ- ent types of sentiment knowledge including senti- ment words, word polarity and aspect-sentiment pairs. The importance of these knowledge has been verified by tasks at different level, for instance, sentence-level sentiment classification (Taboada et al., 2011; Shin et al., 2017; Lei et al., …

Content truncated to 500 words