Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation Zejiang Hou Princeton University, USA∗ zejiangh@princeton.edu Julian Salazar Amazon AWS AI, USA julsal@amazon.com George Polovets Amazon AWS AI, USA polovg@amazon.com Abstract Large pretrained language models (PLMs) are often domain- or task-adapted via fine- tuning or prompting. Finetuning requires modifying all of the parameters and having enough data to avoid overfitting while prompt- ing requires no training and few examples but limits performance. Instead, we pre- pare PLMs for data- and parameter-efficient adaptation by learning to learn the differ- ence between general and adapted PLMs. This difference is expressed in terms of model weights and sublayer structure through our proposed dynamic low-rank reparame- terization and learned architecture controller. Experiments on few-shot dialogue comple- tion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and perfor- mance over direct finetuning or preparation via domain-adaptive pretraining. Ablations show our task-adaptive reparameterization (TARP) and model search (TAMS) components indi- vidually improve on other parameter-efficient transfer like adapters and structure-learning methods like learned sparsification. 1 Introduction Finetuning large pretrained language models (PLMs) on task-specific supervised data has be- come the default strategy to produce performant models for various NLP tasks (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2019, inter alia), provided a task has enough train- ing data to be adapted to without overfitting. For few-shot tasks, very large PLMs like the 175B-parameter GPT-3 (Brown et al., 2020) do surprisingly well without training using prompts, where task-specific examples (xj, yj) are pre- sented as text to condition the PLM before a test input xtest is given. Our work considers an important middle ground: minimizing the com- putational cost of finetuning while improving ∗Work done during an internship at Amazon AWS AI. on its performance in low-resource and few- shot settings. In general, self-supervised objectives used for PLMs assume little about the nature of downstream tasks. Earlier works suggested that task-awareness is unnecessary for PLMs of suf- ficient scale; for example, Raffel et al. (2020) found that multi-task learning underperformed pretrain-finetune for the largest T5 models on multi-format question answering. However, Gururangan et al. (2020) showed that fur- ther pretraining on unlabeled text from the downstream task (task-adaptive pretraining, or TAPT) or a related domain (DAPT) consistently improved adaptation performance. Aghajanyan et al. (2021a) revisited Raffel et al. and found that by greatly improving the number and bal- ance of tasks, one can utilize a multitask objective after pretraining and achieve gains in proportion to the number of tasks. As for even larger mod- els, Brown et al. (2020) argue that the impressive few-shot prompting ability of GPT-3 comes from ‘‘implicit’’ meta-learning (Schmidhuber, 1987; Bengio et al., 1990) which they term in-context learning, where the outer loop is performed by self-supervised pretraining, and the inner loop is performed by forward passes on implicit examples in unlabeled texts. These works motivate that exposure to broad information about downstream tasks remains useful in preparing a large PLM for adaptation. Hence, we …

Content truncated to 500 words