Self-Instruct: Aligning Language Models with Self-Generated Instructions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13484–13508 July 9-14, 2023 ©2023 Association for Computational Linguistics SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions Yizhong Wang♣ Yeganeh Kordi♢ Swaroop Mishra♡ Alisa Liu♣ Noah A. Smith♣+ Daniel Khashabi♠ Hannaneh Hajishirzi♣+ ♣University of Washington ♢Tehran Polytechnic ♡Arizona State University ♠Johns Hopkins University +Allen Institute for AI yizhongw@cs.washington.edu Abstract Large “instruction-tuned” language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to general- ize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the general- ity of the tuned model. We introduce SELF- INSTRUCT, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates in- structions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute im- provement over the original model on SUPER- NATURALINSTRUCTIONS, on par with the performance of InstructGPT001,1 which was trained with private user data and human anno- tations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tun- ing GPT3 with SELF-INSTRUCT outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT001. SELF-INSTRUCT provides an almost annotation-free method for aligning pretrained language models with in- structions, and we release our large synthetic dataset to facilitate future studies on instruction tuning.2 1 Introduction The recent NLP literature has witnessed a tremen- dous amount of activity in building models that can 1Unless otherwise specified, our comparisons are with text-davinci-001 engine. We focus on this engine since it is the closest to our experimental setup: supervised finetuning with human demonstrations. The newer engines are more pow- erful, though use more data (e.g., code or latest user queries) or algorithms (e.g., PPO) that are difficult to compare with. 2Code and data are available at https://github.com/ yizhongw/self-instruct Instruction: Write a letter from the perspective of a cat. Input: Null Output: Dear [Owner], I am writing to you today because I have a problem. You see, I’ve been living with you for the past [Number of Years] and I love it here. The food is great, the bed is comfy, and there are plenty of toys to play with. But lately, I’ve noticed that you’re not as affectionate towards me as you used to be. In fact, sometimes when I try to cuddle up next to you on the couch, you push me away ... Instruction: I am looking for a job and I need to fill out an application form. Can you please help me complete it? Input: Application Form: Name:_________ Age:__________ Sex:________ Phone Number:______ Email Address:________ Education:___________________ ... Output: Name: John Doe Age: 25 Sex: Male Phone Number: ... Instruction: How to write a …

Content truncated to 500 words