On Measuring the Intrinsic Few-Shot Hardness of Datasets

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3955 - 3963 December 7-11, 2022 ©2022 Association for Computational Linguistics On Measuring the Intrinsic Few-Shot Hardness of Datasets Xinran Zhao⋆ Shikhar Murty⋆ Christopher D. Manning Computer Science Department, Stanford University {xzhaoar,smurty,manning}@cs.stanford.edu Abstract While advances in pre-training have led to dra- matic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in datasets. In particular, given a new dataset and a pre-trained model, what properties of the dataset make it few-shot learnable and are these properties independent of the specific adapta- tion techniques used? We consider an exten- sive set of recent few-shot learning methods, and show that their performance across a large number of datasets is highly correlated, show- ing that few-shot hardness may be intrinsic to datasets, for a given pre-trained model. To esti- mate intrinsic few-shot hardness, we then pro- pose a simple and lightweight metric called Spread that captures the intuition that few- shot learning is made possible by exploiting feature-space invariances between training and test samples. Our metric better accounts for few-shot hardness compared to existing notions of hardness, and is ~8–100x faster to compute. 1 Introduction A growing body of recent work has shown im- pressive advances in few-shot adaptation of pre- trained transformers (Radford et al., 2019; Schick and Schütze, 2020; Brown et al., 2020; Karimi Ma- habadi et al., 2021; Liu et al., 2021a, among oth- ers). Despite this progress, there is no concrete understanding of when and why few-shot learning may be successful for a given pre-trained model. Indeed, no free lunch style arguments necessitate the existence of tasks that are not few-shot learn- able by a given pre-trained model, regardless of the adaptation method, and in practice, very simi- lar datasets exhibit varying levels of success when state-of-the-art few-shot adaptation methods are applied (Fig. 1a). ⋆ Equal Contribution Method D1 D2 LMBFF 45.3 -0.4 NullPrompts 43.0 -5.7 BitFit 46.3 -3.5 AdaPET 44.9 -0.3 P-Tuning 46.3 0.3 Few-shot (Avg) 45.2 -2 Full Fine-tuning 45.3 35 D1 : NLI Sentiment D2 : NLI Paraphrase context: When asked about the movie, Cannon said, 'Awful. '. hypothesis: Cannon liked the movie context: When asked about the movie, Ayden said, 'Excellent film. '. hypothesis: Ayden liked the movie context: someone chronicled that a particular thing happened hypothesis: that thing might or might not have happened context: a particular person was satisfied to have a particular thing. hypothesis: that person didn't have that thing Train Set Train Set Test Set D1: Low Spread, Low Few-shot hardness Test Set D2: High Spread, High Few-shot hardness Finetuning Improvement (a) (b) Figure 1: (a) Fine-tuning a model on D1 (D2) on the entire dataset leads to a +45.3 (+35) point improvement over a random baseline. However, most methods are successful at few-shot adaptation on D1 with an average improvement of ~45 points over a random baseline, while few-shot adaptation is unsuccessful on D2. (b) We observe that features …

Content truncated to 500 words