Quantifying Adaptability in Pre-trained Language Models with 500 Tasks

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4696 - 4715 July 10-15, 2022 ©2022 Association for Computational Linguistics Quantifying Adaptability in Pre-trained Language Models with 500 Tasks Belinda Z. Li MIT bzl@mit.edu Jane Yu Meta AI janeyu@fb.com Madian Khabsa Meta AI mkhabsa@fb.com Luke Zettlemoyer Meta AI lsz@fb.com Alon Halevy Meta AI ayh@fb.com Jacob Andreas MIT jda@mit.edu Abstract When a neural language model (LM) is adapted to perform a new task, what as- pects of the task predict the eventual per- formance of the model? In NLP, system- atic features of LM generalization to individ- ual examples are well characterized, but sys- tematic aspects of LM adaptability to new tasks are not nearly as well understood. We present a large-scale empirical study of the fea- tures and limits of LM adaptability using a new benchmark, TASKBENCH500, built from 500 procedurally generated sequence model- ing tasks. These tasks combine core aspects of language processing, including lexical seman- tics, sequence processing, memorization, log- ical reasoning, and world knowledge. Using TASKBENCH500, we evaluate three facets of adaptability, finding that: (1) adaptation pro- cedures differ dramatically in their ability to memorize small datasets; (2) within a subset of task types, adaptation procedures exhibit com- positional adaptability to complex tasks; and (3) failure to match training label distributions is explained by mismatches in the intrinsic dif- ficulty of predicting individual labels. Our ex- periments show that adaptability to new tasks, like generalization to new examples, can be systematically described and understood, and we conclude with a discussion of additional as- pects of adaptability that could be studied us- ing the new benchmark. 1 Introduction Much of the recent research effort in NLP has shifted from training task-specific models to adapt- ing pre-trained language models (LMs) by fine- tuning their parameters or input prompts for down- stream tasks (Devlin et al., 2019; Raffel et al., 2020; Li and Liang, 2021; Lester et al., 2021). This paradigm is general, in the sense that a large number of distinct NLP tasks benefit from pre- training (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020). But many questions about the nature and limits of LM adaptation remain unan- swered. For example: given a new task, can we pre- dict how quickly (and how effectively) pre-trained LMs can be adapted to perform it? From among the variety of different adaptation techniques (e.g. fine- tuning or prompt-tuning), can we predict which one will be most effective? Today, new pre-training and adaptation schemes are evaluated using small suites of curated tasks, typically featuring classification, textual inference, and question answering (Wang et al., 2018, 2019). These benchmarks have been extremely successful in identifying new tools for adaptation, but they are poorly suited for answering larger, structural questions like the ones above. We present a large-scale study of LM adaptabil- ity using a new suite of benchmark tasks called TASKBENCH500.1 TASKBENCH500 consists of 500 procedurally generated tasks involving lexi- …

Content truncated to 500 words