Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4019–4028 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4019 Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis 12Chunning Du, 12Haifeng Sun, 12Jingyu Wang∗, 12Qi Qi, 12Jianxin Liao 1State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China 2EBUPT Information Technology Co., Ltd., Beijing 100191, China {duchunning,sunhaifeng 1,wangjingyu,qiqi}@ebupt.com Abstract Cross-domain sentiment classification aims to address the lack of massive amounts of la- beled data. It demands to predict sentiment polarity on a target domain utilizing a clas- sifier learned from a source domain. In this paper, we investigate how to efficiently apply the pre-training language model BERT on the unsupervised domain adaptation. Due to the pre-training task and corpus, BERT is task- agnostic, which lacks domain awareness and can not distinguish the characteristic of source and target domain when transferring knowl- edge. To tackle these problems, we design a post-training procedure, which contains the target domain masked language model task and a novel domain-distinguish pre-training task. The post-training procedure will encour- age BERT to be domain-aware and distill the domain-specific features in a self-supervised way. Based on this, we could then con- duct the adversarial training to derive the en- hanced domain-invariant features. Extensive experiments on Amazon dataset show that our model outperforms state-of-the-art methods by a large margin. The ablation study demon- strates that the remarkable improvement is not only from BERT but also from our method. 1 Introduction Sentiment analysis aims to automatically identify the sentiment polarity of the textual data. It is an essential task in natural language processing with widespread applications, such as movie reviews and product recommendations. Recently, deep net- works have significantly improved the state-of-the- art in sentiment analysis. However, training deep networks is highly depended on a large amount of labeled training data which is time-consuming and requires expensive manual labeling. Thus, there is a strong need to leverage or reuse rich labeled ∗Corresponding author. data from a different but related source domain. Cross-domain sentiment analysis, which transfers the knowledge learned from source domain to a new target domain, becomes a promising direction. The main challenge of cross-domain sentiment analysis is domain discrepancy due to different expression of the user’s emotion across domains. To address the problem, a wide-used approach is designed to extract domain invariant features, which means that the distributions of features from the source domain and target domain are simi- lar (Zellinger et al., 2017; Persello and Bruzzone, 2016; Ganin et al., 2016; Yu and Jiang, 2016a). One effective way to obtain domain-invariant features is adversarial training(Ganin et al., 2016; Li et al., 2017; Zheng et al., 2019). Specifically, A domain discriminator is learned by minimizing the classifi- cation error of distinguishing the source from the target domains, while a deep classification model learns transferable representations that are indistin- guishable by the domain discriminator. Very recently, pre-trained language models have shown to be effective for improving many language …

Content truncated to 500 words