How many data points is a prompt worth?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636 June 6–11, 2021. ©2021 Association for Computational Linguistics 2627 How Many Data Points is a Prompt Worth? Teven Le Scao Hugging Face teven@huggingface.co Alexander M. Rush Hugging Face sasha@huggingface.co Abstract When fine-tuning pretrained models for classi- fication, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task- specific guidance, which is beneficial in low- data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this ben- efit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks. 1 Introduction The main paradigm for adapting pretrained mod- els for classification (Radford, 2018; Dong et al., 2019; Devlin et al., 2018) is fine-tuning via an ex- plicit classifier head. However, an alternative ap- proach has arisen: adapting the pretrained language model directly as a predictor through autoregres- sive text generation (Radford et al., 2019) or com- pletion of a cloze task (Trinh and Le, 2018). This method is notably used in T5 fine-tuning (Raffel et al., 2019) leading to state-of-the-art results on the SuperGLUE benchmark (Wang et al., 2019). One argument made for classification by direct language generation is that it allows us to pick cus- tom prompts for each task (McCann et al., 2018). While this approach can be used for zero-shot classification (Puri and Catanzaro, 2019) or prim- ing (Brown et al., 2020), it can also be used in fine-tuning to provide extra task information to the classifier, especially in the low-data regime (Schick and Schütze, 2020a,b). Code available at https://github.com/TevenLeScao/pet If this argument is indeed true, it is natural to ask how it impacts the sample efficiency of the model, or more directly, how many data points is a prompt worth? As with many low-data and pretraining- based problems, this question is complicated by the fine-tuning setup, training procedure, and prompts themselves. We attempt to isolate these variables through diverse prompts, multiple runs, and best practices in low-training data fine-tuning. We in- troduce a metric, the average data advantage, for quantifying the impact of a prompt in practice. Our experiments find that the impact of task- targeted prompting can nicely be quantified in terms of direct training data, and that it varies over the nature of different tasks. On MNLI (Williams et al., 2018), we find that using a prompt con- tributes approximately 3500 data points. On Su- perGLUE, it adds approximately 280 data points on RTE (Dagan et al., 2005) and up to 750 on BoolQ (Clark et al., 2019). In low- to medium-data settings, this advantage can be a real …

Content truncated to 500 words