Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2438 - 2447 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis Hui Wu12 and Xiaodong Shi123∗ 1Department of Artificial Intelligence, School of Informatics, Xiamen University, China 2National Institute for Data Science in Health and Medicine, Xiamen University, China 3Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China huistudent@stu.xmu.edu.cn, mandel@xmu.edu.cn Abstract Cross-domain sentiment analysis has achieved promising results with the help of pre-trained language models. As GPT-3 appears, prompt tuning has been widely explored to enable bet- ter semantic modeling in many natural lan- guage processing tasks. However, directly using a fixed predefined template for cross- domain research cannot model different distri- butions of the [MASK] token in different do- mains, thus making underuse of the prompt tuning technique. In this paper, we propose a novel Adversarial Soft Prompt Tuning method (AdSPT) to better model cross-domain sen- timent analysis. On the one hand, AdSPT adopts separate soft prompts instead of hard templates to learn different vectors for differ- ent domains, thus alleviating the domain dis- crepancy of the [MASK] token in the masked language modeling task. On the other hand, AdSPT uses a novel domain adversarial train- ing strategy to learn domain-invariant repre- sentations between each source domain and the target domain. Experiments on a publicly available sentiment analysis dataset show that our model achieves new state-of-the-art results for both single-source domain adaptation and multi-source domain adaptation. 1 Introduction In recent years, with the emergence of a series of large-scale pre-trained language models (PLMs), such as GPT (Radford et al., 2018, 2019), BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019), fine-tuning PLMs has achieved promising results on a wide range of natural language pro- cessing (NLP) tasks. However, as PLMs become larger and larger, fine-tuning larger PLMs becomes more challenging in most real-world applications. More recently, Brown et al. (2020) show that de- signing task descriptions (a.k.a. prompts) can make accurate predictions without updating any of the ∗Corresponding author. Reading it can learn calculus better . Watching it is like being on the scene . It was . [MASK] It was . [MASK] MLM head MLM head Label Words useful benificial good ... ... Label Words real amazing good ... ... Template A Book Review A Video Review Template ? ? Figure 1: How domain discrepancy affects prompt tun- ing. Examples of a book review on the top and a video review on the bottom. parameters of GPT-3 (which has 175B parameters). This inspires a new PLM-tuning method named “prompt tuning”. Such prompt tuning method has achieved state-of-the-art results on text clas- sification and natural language inference (Schick and Schütze, 2020; Schick et al., 2020; Gao et al., 2020), relation classification (Han et al., 2021), and natural language generation (Li and Liang, 2021). It is common to use …

Content truncated to 500 words