UserAdapter: Few-Shot User Learning in Sentiment Analysis

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1484–1488 August 1–6, 2021. ©2021 Association for Computational Linguistics 1484 UserAdapter: Few-Shot User Learning in Sentiment Analysis Wanjun Zhong1∗, Duyu Tang2, Jiahai Wang1, Jian Yin1 and Nan Duan2 1 The School of Computer Science and Engineering, Sun Yat-sen University. Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R.China 2 Microsoft Research {zhongwj25@mail2, wangjiah@mail,issjyin@mail}.sysu.edu.cn {dutang,nanduan}@microsoft.com Abstract Adapting a model to a handful of personal- ized data is challenging, especially when it has gigantic parameters, such as a Transformer- based pretrained model. The standard way of fine-tuning all the parameters necessitates storing a huge model for each user. In this work, we introduce a lightweight approach dubbed UserAdapter, which clamps hundred millions of parameters of the Transformer model and optimizes a tiny user-specific vec- tor. We take sentiment analysis as a test bed, and collect datasets of reviews from Yelp and IMDB respectively. Results show that, on both datasets, UserAdapter achieves better accu- racy than the standard fine-tuned Transformer- based pre-trained model. More importantly, UserAdapter offers an efficient way to produce a personalized Transformer model with less than 0.5% parameters added for each user. 1 Introduction Having a bespoke model by only seeing a few data of a user is increasingly important given its mer- its of producing customized service and protecting user privacy. In this work, we study the learning of personalized model based on Transformer-based pretrained models (Devlin et al., 2018; Liu et al., 2019), which dominate a wide range of natural lan- guage understanding problems. A standard way is fine-tuning the whole parameters. However, this is unacceptable in practice because it would result in storing a model with hundreds of millions of parameters for each user. An alternative method is in-context learning, which is adopted in GPT-3 (Brown et al., 2020). A few examples are pro- vided as context to the pretrained model and no fine-tuning is needed. However, limited by the ∗ Work done while this author was an intern at Microsoft Research. bounded-length context of Transformer, it cannot make full use of the training instances that exceed the context window. In this work, we introduce UserAdapter, a lightweight method that learns personalized model in a few-shot learning scenario. Our work is in- spired by the recent progress on lightweight fine- tuning (Houlsby et al., 2019; Wang et al., 2020; Li and Liang, 2021), where a small number of task-specific parameters are the only trainable ones, while the dominant parameters of Transformer are fixed. In UserAdapter, each user is represented as a continuous vector, and such vector works as a virtual “prefix” token that steers the representa- tions produced by Transformer. When adapting an existing model to a few datapoints of a new user, we clamp the parameters of Transformer and only need to train the tiny user vector. Even if taking the parametrization strategy into account (detailed in Section 3.2), UserAdapter adds less than 0.5% parameters for each user. As a case study, we conduct …

Content truncated to 500 words