Making Pretrained Language Models Good Long-tailed Learners

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3298 - 3312 December 7-11, 2022 ©2022 Association for Computational Linguistics Making Pretrained Language Models Good Long-tailed Learners Chen Zhang , Lei Ren , Jingang Wang , Wei Wu , Dawei Song Beijing Institute of Technology {czhang,dwsong}@bit.edu.cn Meituan NLP {wangjingang02,wuwei30}@meituan.com renlei_work@163.com Abstract Prompt-tuning has shown appealing perfor- mance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hy- pothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demon- strate that prompt-tuning makes pretrained lan- guage models at least good long-tailed learn- ers. For intuitions on why prompt-tuning can achieve good performance in long-tailed clas- sification, we carry out in-depth analyses by progressively bridging the gap between prompt- tuning and commonly used finetuning. The summary is that the classifier structure and pa- rameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we ver- ify the applicability of our finding to few-shot classification.1 1 Introduction Pretrained language models (PLMs) with CLS- tuning (i.e., finetuning a PLM by applying a clas- sifier head over the [CLS] representation) have achieved strong performance in a wide range of downstream classification tasks (Devlin et al., 2019; Liu et al., 2019a; Raffel et al., 2020; Wang et al., 2019b). However, they have been less promising in the long-tailed scenario (Li et al., 2020). The long-tailed scenario is different from common sce- narios due to the long tail phenomenon exhibited in the class distribution, as illustrated in Figure 1. The long-tailed class distribution prevents PLMs from achieving good performance, especially in authors.Jingang Wang and Dawei Song are the corresponding 1Good long-tailed learners are abbreviated as GLEE. Code and data are available at https://github.com/ GeneZC/Glee. 0 20 40 60 80 100 120 Class index (from head to tail). 0 250 500 750 1000 1250 1500 1750 #Examples. Head Tail Figure 1: An example long-tailed class distribution from IFLYTEK dataset (Xu et al., 2020), where we can distin- guish tail classes from head classes. tail classes that only allow learning with very few examples (dubbed tail bottleneck). Recent advances on Prompt-tuning2 have wit- nessed a surge of making PLMs better few-shot learners (Schick and Schütze, 2021; Gao et al., 2021; Scao and Rush, 2021). Prompt-tuning re- quires PLMs to perform classification in a cloze style, thus is superior to CLS-tuning in two as- pects: 1) it aligns the input structure with that of masked language modeling (MLM); and 2) it re- uses the classifier structure and parameterization from the pretrained MLM head. These two merits of Prompt-tuning equip PLMs to better exploit pre- trained knowledge and hence gain a better few-shot performance than that with CLS-tuning. The success of Prompt-tuning in few-shot sce- narios motivates us to hypothesize that Prompt- tuning can relieve the …

Content truncated to 500 words