Few-shot Adaptation Works with UnpredicTable Data

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1806–1842 July 9-14, 2023 ©2023 Association for Computational Linguistics Few-shot Adaptation Works with UnpredicTable Data Jun Shern Chan1 2 Michael Pieler1 2 Jonathan Jao1 2 Jérémy Scheurer1 2 Ethan Perez1 2 3∗ 1New York University, 2Fund for Alignment Research, 3Anthropic {junshern,perez}@nyu.edu Abstract Prior work on language models (LMs) shows that training on a large number of diverse tasks improves few-shot learning (FSL) perfor- mance on new tasks. We take this to the ex- treme, automatically extracting 413,299 tasks from internet tables - orders of magnitude more than the next-largest public datasets. Finetuning on the resulting dataset leads to improved FSL performance on Natural Lan- guage Processing (NLP) tasks, but not propor- tionally to dataset scale. In fact, we find that narrow subsets of our dataset sometimes out- perform more diverse datasets. For example, finetuning on software documentation from support.google.com raises FSL perfor- mance by a mean of +7.5% on 52 downstream tasks, which beats training on 40 human- curated NLP datasets (+6.7%). Finetuning on various narrow datasets leads to similar broad improvements across test tasks, suggesting that the gains are not from domain adapta- tion but adapting to FSL in general. We do not observe clear patterns between the datasets that lead to FSL gains, leaving open questions about why certain data helps with FSL. 1 Introduction Brown et al. (2020) showed that language models (LMs) learn to perform new tasks from a few exam- ples (“few-shot learning”; FSL). Explicitly training LMs for FSL further improves performance (Min et al., 2021; Chen et al., 2021b), and prior work has found that increasing the size and diversity of training tasks improves generalization to new tasks (Sanh et al., 2021; Aribandi et al., 2021; Agha- janyan et al., 2021a; Wang et al., 2022). We push size and diversity to the extreme by finetuning on a large dataset of automatically-curated FSL tasks, and surprisingly find that certain narrow datasets of tasks (e.g. software documentation) outperform much larger and more diverse datasets. ∗Work done primarily at NYU and FAR. If you want to ... Then ... Report spam Submit a spam report. Get a page or site remove3. Submit a URL removal request. Tell Google to crawl your sK. Request a change in crawl rate. . . . . . . [If you want to ...] Report spam [Then ...] Submit a spam report. Input Input Output [If you want to ...] Get a page or site remov�. Submit a URL removal request. Input Input Output . . . Input . . . } 4) Outperform in few-shot task transfer?! multi-task training with 40 NLP datasets Scrape HTML tables from support.google.com. 1 Convert tables to few-shot tasks. 2 Fine-tune an LM on the generated tasks. 3 4 Figure 1: We convert web tables into FSL tasks, then use these tasks via finetuning to adapt language models for FSL. Unexpected tables lead to strong task trans- fer: finetuning …

Content truncated to 500 words