SCOTT: Self-Consistent Chain-of-Thought Distillation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5546–5558 July 9-14, 2023 ©2023 Association for Computational Linguistics SCOTT: Self-Consistent Chain-of-Thought Distillation Peifeng Wang1∗, Zhengyang Wang2, Zheng Li2, Yifan Gao2, Bing Yin2, Xiang Ren1 1Department of Computer Science, University of Southern California, 2Amazon.com Inc {peifengw,xiangren}@usc.edu, {zhengywa,amzzhe,yifangao,alexbyin}@amazon.com Abstract Large language models (LMs) beyond a cer- tain scale, demonstrate the emergent capability of generating free-text rationales for their pre- dictions via chain-of-thought (CoT) prompt- ing. While CoT can yield dramatically im- proved performance, such gains are only ob- served for sufficiently large LMs. Even more concerning, there is little guarantee that the generated rationales are consistent with LM’s predictions or faithfully justify the decisions. In this work, we propose SCOTT, a faith- ful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger. To form better supervision, we elicit rationales supporting the gold answers from a large LM (teacher) by contrastive decoding, which encourages the teacher to generate to- kens that become more plausible only when the answer is considered. To ensure faithful distillation, we use the teacher-generated ratio- nales to learn a student LM with a counterfac- tual reasoning objective, which prevents the student from ignoring the rationales to make inconsistent predictions. Experiments show that, while yielding comparable end-task per- formance, our method can generate CoT ratio- nales that are more faithful than baselines do. Further analysis suggests that such a model re- spects the rationales more when making deci- sions; thus, we can improve its performance more by refining its rationales. 1 Introduction Large language models (LMs) elicit strong reason- ing capabilities through chain-of-thought (CoT) prompting (Wei et al., 2022b), which asks LMs to generate free-text rationale for explaining their multi-step reasoning. However, CoT prompting does not guarantee that the rationale is consis- tent with the prediction, rendering the rationale ∗This work was done when Peifeng Wang was an intern at Amazon. Zheng Li and Xiang Ren are corresponding authors. Can a Bengal cat survive eating only pancakes? The answer is no. Why? Is material from an aloe plant sometimes enclosed in petroleum-derived products? The answer is yes. Why? A Bengal cat cannot survive eating only pancakes. Aloe is a plant Plants are made of cells. Cells are made of molecules. Molecules are made of atoms. 👎 Error 1 (42%): Do not provide new information. 👎 Error 2 (37%): Do not justify the answer. GPT-3 GPT-3 Figure 1: Vacuous rationales generated by a prompted LM (GPT-3) for StrategyQA. In both types of error cases, LM fails to give rationales consistent with the answers due to hallucination. useless for justifying the model’s behavior. In this work, we present Self-Consistent Chain-Of- Thought DisTillation (SCOTT), a knowledge dis- tillation (KD) method for eliciting faithful CoT rea- soning, where a small student model learns from a large teacher model to generate CoT rationales that are consistent to its own predictions. Existing works (Shridhar et al., 2022; …

Content truncated to 500 words