HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11272–11288 July 9-14, 2023 ©2023 Association for Computational Linguistics HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation Hamish Ivison α Akshita Bhagia α Yizhong Wang ω Hannaneh Hajishirzi αω Matthew Peters α αAllen Institute for AI ωPaul G. Allen School of Computer Science & Engineering, University of Washington {hamishi,akshitab,yizhongw,hanna,matthewp}@allenai.org Abstract Recent NLP models have shown the remark- able ability to effectively generalise ‘zero-shot’ to new tasks using only natural language in- structions as guidance. However, many of these approaches suffer from high computa- tional costs due to their reliance on concate- nating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hy- pernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text en- coder, eliminating the need to include instruc- tions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs dur- ing decoding to further improve performance. HINT models outperform strong state-of-the- art baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of com- pute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning. We release our code pub- licly1. 1 Introduction Large pretrained language models have demon- strated a striking ability to perform new tasks through the use of in-context examples or instruc- tions alone (Brown et al., 2020), or after training on input instances augmented with instructions (Weller et al., 2020; Mishra et al., 2022; Sanh et al., 2022; Wei et al., 2022; Chung et al., 2022; Wang et al., 2022b). This ability allows a single model to 1Our code is available at: https://github.com/allenai/hyper-task-descriptions. HyperEncoder Parameter Generator Encoder Decoder Task Definition: You will be given sentences in which your task is to recognize the name of a person. Encoded Instruction Instance Input: In "House of X", Moira MacTaggert was revealed to be a mutant with the ability to reincarnate. Encoded Instance Generated Adapters Generated Prefixes Output: Moira MacTaggert (1) Hypernetwork (2) Underlying Model Encoded Instr. Figure 1: Overview of HINT. (1) We feed an instruction into a HyperEncoder to produce an encoded instruc- tion, use it to generate prefix and adapter weights, and then insert them into the underlying model. (2) We run the underlying model encoder as usual and optionally concatenate the encoded input with the previously en- coded instruction, before running the underlying model decoder to generate the answer. We only use the hyper- network once per task. adapt to many tasks where training data is difficult to collect or task-specific fine-tuning is impractical (i.e., ‘zero-shot’ …

Content truncated to 500 words