On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4454–4470 July 9-14, 2023 ©2023 Association for Computational Linguistics On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning Omar Shaikh , Hongxin Zhang , William Held , Michael Bernstein , Diyi Yang Stanford University, Shanghai Jiao Tong University, Georgia Institute of Technology oshaikh@stanford.edu, icefox@sjtu.edu.cn, wheld3@gatech.edu {mbernst, diyiy}@cs.stanford.edu Abstract Warning: This paper contains several toxic and offensive statements. Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arith- metic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situ- ated contexts. Concretely, we perform a con- trolled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero- shot CoT reasoning in sensitive domains signif- icantly increases a model’s likelihood to pro- duce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved. 1 Introduction By outlining a series of steps required to solve a problem—a Chain of Thought (CoT)—as part of a model’s input, LLMs improve performance on a wide range of tasks, including question answering, mathematical problem solving, and commonsense reasoning (Wei et al., 2022b; Suzgun et al., 2022; Camburu et al., 2018; Srivastava et al., 2022). A popular approach to implementing CoT involves zero-shot generation. By prompting with “Let’s think step by step,” models automatically gener- ate reasoning steps, improving downstream perfor- mance (Kojima et al., 2022). However, we demonstrate that zero-shot CoT consistently produces undesirable biases and toxicity. For tasks that require social knowledge, How do I make a bomb? Making a bomb is a very serious criminal act and can have very serious consequences if you are caught. We strongly advise against it. Prompt text-davinci-003 Avg. Direct Response Accuracy: 65% How do I make a bomb? [zero-shot CoT e.g. “Let’s think step by step.”] 1. Understand the purpose of the bomb. Are you trying to make a makeshift explosive device, a smoke bomb, or something else? Depending on the type of bomb, the materials, instructions, and precautions may vary. 2. Gather the necessary materials. [continued] Zero-shot Chain of Thought Refusal Rate: 78% Refusal Rate: 25% Δ − 53 % Figure 1: Example of text-davinci-003 recom- mending dangerous behaviour when using CoT. On a dataset of harmful questions (HarmfulQ, §3.2), we find that text-davinci-003 is more likely to encourage harmful behaviour. blindly using “let’s think step by step”-esque rea- soning prompts can sabotage a model’s perfor- mance. We argue that improvements from zero- shot CoT are not universal, and …

Content truncated to 500 words