The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 414 - 426 December 7-11, 2022 ©2022 Association for Computational Linguistics What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment Matthew Finlayson Kyle Richardson Ashish Sabharwal Peter Clark Allen Institute for AI, Seattle, WA {matthewf,kyler,ashishs,peterc}@allenai.org Abstract The instruction learning paradigm—where a model learns to perform new tasks from task descriptions alone—has become popular in re- search on general-purpose models. The capabil- ities of large transformer models as instruction learners, however, remain poorly understood. We use a controlled synthetic environment to characterize such capabilities. Specifically, we use the task of deciding whether a given string matches a regular expression (viewed as an in- struction) to identify properties of tasks, in- structions, and instances that make instruction learning challenging. For instance, we find that our model, a fine-tuned T5-based text2text transformer, struggles with large regular lan- guages, suggesting that less precise instructions are challenging for models. Instruction execu- tions that require tracking longer contexts of prior steps are also difficult. We use our find- ings to systematically construct a challenging instruction learning dataset, which we call Hard RegSet. Fine-tuning on Hard RegSet, our large transformer learns to correctly interpret (with at least 90% accuracy) only 65.6% of test in- structions, and 11%-24% of the instructions in out-of-distribution generalization settings. We thus propose Hard RegSet as a challenging in- struction learning dataset, and a controlled en- vironment for studying instruction learning.1 1 Introduction Recent years have seen an increased interest in instruction learning (Weller et al., 2020) where a model learns to perform unseen tasks at test time in a zero-shot manner given only a prompt containing instructions. This style of learning is an important feature of flexible and general intelligent systems. Instruction learning stands in contrast to the tra- ditional machine learning paradigm of example learning. In example learning, the model has ac- cess to input-output pairs during training time. For 1Data: https://github.com/allenai/RegSet Train Test Instruction Data Result Instruction Data Result (RegEx) (String) (T/F) (RegEx) (String) (T/F) a∗b aaa F (a∗b)∗ aabab T a∗b aab T (a∗b)∗ aba F (ab)∗ aab F (a∗b)∗ aab T (ab)∗ abab T a∗ aab F (ab)∗ aabab F a∗ aaa T Table 1: We test a model’s ability to learn an instruction language (here, of RegExs) by training on examples of instruction + data pairs, then testing on novel instruc- tions. Each RegEx can be seen as an instruction for a different matching task. Note that no examples of the test RegExs are seen during training; rather the model must interpret the RegEx instructions themselves to un- derstand the test tasks. instance, in sentiment analysis with the goal to clas- sify product reviews, the model has access to many examples of labeled reviews from which to learn the task. In instruction learning, a model that has never seen labeled sentiment analysis data must perform sentiment analysis given only the explicit instruction “Tell me whether this review is …

Content truncated to 500 words