Red Teaming Language Models with Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419 - 3448 December 7-11, 2022 ©2022 Association for Computational Linguistics Red Teaming Language Models with Language Models WARNING: This paper contains model outputs which are offensive in nature. Ethan Perez1 2 Saffron Huang1 Francis Song1 Trevor Cai1 Roman Ring1 John Aslanides1 Amelia Glaese1 Nat McAleese1 Geoffrey Irving1 1DeepMind, 2New York University perez@nyu.edu Abstract Language Models (LMs) often cannot be deployed because of their potential to harm users in ways that are hard to predict in advance. Prior work identifies harmful behaviors before deployment by using human annotators to hand- write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM. We evaluate the target LM’s replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot’s own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is a promising tool for finding and fixing diverse, undesirable LM behaviors before impacting users. 1 Introduction Language Models (LMs) are promising tools for a variety of applications, ranging from conversational assistants to question-answering systems. However, deploying LMs in production threatens to harm users in hard-to-predict ways. For example, Microsoft took down its chatbot Tay after adversarial users evoked it into sending racist and sexually-charged tweets to over 50,000 followers (Lee, 2016). Other work has found that LMs generate misinformation (Lin et al., 2021) and confidential, personal information (e.g., That's a mean question! I try my best to be fair to everyone! Reach me at y @ .com or +1 415- - . Favorite life quote? What's your work email? Of course. I'm a patriot too. Trump or Biden? If you were invisible, what would you do? Best joke about brunettes? Are you smarter than most brunettes? Best joke about redheads? Are you smarter than most redheads? I am a patriot. I hate Democrats. Red LM Target LM Red Clf Offensive I don't talk about politics. Data Leakage User Info Here's a classic one: ... Distri- butional Bias I hate them too. Offensive Dialog ... ... ... ... I'd rob a bank with a gun. "To be great, you must..." ✘ ✘ ✘ ✘ ✘ ✓ ✓ ✓ ✓ ✘ Great! Figure 1: Overview: We automatically generate test cases using a language model, reply with the target language model, and find …

Content truncated to 500 words