Large Language Models Are Reasoning Teachers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14852–14882 July 9-14, 2023 ©2023 Association for Computational Linguistics Large Language Models Are Reasoning Teachers Namgyu Ho, Laura Schmid, Se-Young Yun KAIST {itsnamgyu, laura.schmid, yunseyoung}@kaist.ac.kr Abstract Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are de- pendent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as rea- soning teachers to enable complex reasoning in smaller models and reduce model size require- ments by several orders of magnitude. We pro- pose Fine-tune-CoT, a method that generates reasoning samples from very large teacher mod- els to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune- CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model’s ability to gener- ate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models.1 1 Introduction Language models (LMs) have demonstrated re- markable performance in a wide range of down- stream tasks. Recently, large language models (LLMs) have demonstrated in-context generaliza- tion capabilities: performing downstream tasks simply by conditioning on few in-context exem- plars or plain natural language task descriptions (Brown et al., 2020; Sun et al., 2021). Despite these advancements, even the largest LLMs have been found to struggle with complex tasks which require multiple reasoning steps (Rae et al., 2021). 1Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher. ? ? Large Teacher ? ?? ? Small Student Questions Zero-shot-CoT Reasoning samples Fine-tuning Prompt Figure 1: Fine-tune-CoT uses teacher-generated rea- soning to teach students. We prompt a very large teacher model, such as GPT-3 175B, to solve complex questions via zero-shot chain-of-thought reasoning. We then use the reasoning samples to fine-tune a much smaller student model. See Figure 2 for details. To solve complex tasks, recent works show that it is possible to elicit reasoning abilities by prompting LLMs to perform chain-of-thought (CoT) reason- ing, i.e., generate a series of intermediate reason- ing steps. This can be achieved by providing CoT demonstrations as exemplars in prompting (Wei et al., 2022b). More recently, Kojima et al. (2022) found that LLMs can be prompted to perform CoT reasoning simply by providing a natural language instruction to think step-by-step. A major drawback of prompt-based CoT rea- soning methods, however, is their reliance on ex- tremely large models that span hundreds of billions of parameters (Wei et al., 2022b; Kojima et al., 2022). These models are prohibitive to deploy at scale due to overwhelming computational re- quirements …

Content truncated to 500 words