Making Language Models Better Reasoners with Step-Aware Verifier

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5315–5333 July 9-14, 2023 ©2023 Association for Computational Linguistics Making Large Language Models Better Reasoners with Step-Aware Verifier Yifei Li1,2∗, Zeqi Lin2, Shizhuo Zhang2, Qiang Fu2, Bei Chen2, Jian-Guang Lou2, Weizhu Chen2 1 National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University 2 Microsoft Corporation {yifeili, zeqi.lin, v-shizzhang, qifu, bei.chen, jlou, wzchen}@microsoft.com liyifei@stu.pku.edu.cn Abstract Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face diffi- culties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To im- prove their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a sig- nificant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this pa- per, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DIVERSE has three main components: first, it generates diverse prompts to explore dif- ferent reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step indi- vidually instead of the whole chain. We eval- uate DIVERSE on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% → 83.2%). 1 Introduction Large pretrained language models (PLMs) have shown remarkable performance on various natural language processing tasks, either by few-shot learn- ing with prompts (Radford et al., 2019; Le Scao and Rush, 2021; Jin et al., 2022) or by fine-tuning (Houlsby et al., 2019; Hu et al., 2021; He et al., 2022). However, despite the increasing size and capacity of PLMs such as GPT-3 with 175B param- eters (Brown et al., 2020) and PaLM with 540B parameters (Chowdhery et al., 2022), their reason- ing abilities are still limited and often require mul- ∗Work was done during an internship at Microsoft Re- search Asia. Figure 1: Our proposed method, DIVERSE (Diverse Verifier on Reasoning Step). tiple steps to produce correct answers, especially for tasks involving arithmetic, commonsense, or inductive reasoning (Cobbe et al., 2021). Recent works (Wei et al., 2022; Zhou et al., 2022; Kojima et al., 2022; Lampinen et al., 2022) have demonstrated that PLMs possess some latent rea- soning capabilities, but they need carefully de- signed prompts to activate them. For instance, Wei et al. (2022) proposed chain-of-thought reasoning, which inserts multi-step reasoning paths before gen- erating the final answers, and achieved significant improvement on the GSM8K arithmetic benchmark (Cobbe et al., 2021). Wang et al. (2022c) further introduced a voting mechanism to select the most consistent answer among different reasoning paths, and achieved state-of-the-art results on several rea- soning benchmarks using …

Content truncated to 500 words