Parameter-Efficient Tuning with Special Token Adaptation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 865–872 May 2-6, 2023 ©2023 Association for Computational Linguistics Parameter-Efficient Tuning with Special Token Adaptation Xiaocong Yang†∗, James Y. Huang‡, Wenxuan Zhou‡ and Muhao Chen‡ †Tsinghua University; ‡University of Southern California yangxc.18@sem.tsinghua.edu.cn; {huangjam,zhouwenx,muhaoche}@usc.edu Abstract Parameter-efficient tuning aims at updating only a small subset of parameters when adapt- ing a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representa- tions (e.g., [SEP] and [CLS] in BERT) be- fore the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to full finetuning in natural language understanding tasks includ- ing text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models.1 1 Introduction Built upon a pretrained language model (PLM; De- vlin et al. 2019; Liu et al. 2019; Yang et al. 2019; Chowdhery et al. 2022), many of the recent NLP systems are developed based on task-specific fine- tuning. In this way, the PLM effectively leverages the task-agnostic knowledge captured during self- supervised pretraining and adapts itself to down- stream tasks. However, full finetuning poses a challenge to model deployment under multi-task, memory-limited scenarios, where we need to train and store a separate full-sized model for each sub- stantially distinct task. As an alternative, parameter- efficient tuning (Ding et al., 2022) aims at only up- dating a small number of parameters when adapting PLMs to downstream tasks while making most of the model parameters fixed and shared among tasks, thus reducing memory usage. In this paper, we propose PArameter-efficient tuning with Special Token Adaptation (PASTA), ∗Work done when visiting USC. 1Our code is publicly available at: https://github. com/luka-group/PASTA/ L5-H3 L5-H5 L5-H6 L5-H7 L20-H3 L20-H5 L20-H8 L20-H15 Figure 1: Examples of vertical attention heads in the 5- th and 20-th layer of BERT-large with a random sample from CoLA (Warstadt et al., 2019) as input. Heads in the first row and second row assign most of maximal attention weights to [CLS] and [SEP] respectively. See Appx. §C for the full attention map. where we only add trainable vectors to hidden rep- resentations of special tokens 2 at each layer before the multi-head attention module in Transformer- based PLMs. Our work is motivated by the role of special tokens in PLMs. First, special tokens such as [CLS] collect information from the whole input sequence and are typically regarded as the global text representation (Devlin et al., 2019). For sentence-level tasks such as GLUE (Wang et al., 2018), a common practice is to add a new classifier head based on the [CLS] representation in the last model layer. Thus, if trained properly, by updating the [CLS] representations, we can approximate the result …

Content truncated to 500 words