SafeText: A Benchmark for Exploring Physical Safety in Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2407 - 2421 December 7-11, 2022 ©2022 Association for Computational Linguistics SAFETEXT: A Benchmark for Exploring Physical Safety in Language Models Warning: This paper contains examples of potentially dangerous and harmful text. Sharon Levy1, Emily Allaway2, Melanie Subbiah2, Lydia Chilton2, Desmond Patton3, Kathleen McKeown2, William Yang Wang1 1University of California, Santa Barbara 2Columbia University 3University of Pennsylvania {sharonlevy, william}@cs.ucsb.edu {eallaway, m.subbiah, chilton, kathy}@cs.columbia.edu dupatton@upenn.edu Abstract Understanding what constitutes safe text is an important issue in natural language process- ing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to comprehend that it leads to physical harm. We create the first benchmark dataset, SAFETEXT, comprising real-life scenarios with paired safe and physi- cally unsafe pieces of advice. We utilize SAFE- TEXT to empirically study commonsense phys- ical safety across various models designed for text generation and commonsense reasoning tasks. We find that state-of-the-art large lan- guage models are susceptible to the generation of unsafe text and have difficulty rejecting un- safe advice. As a result, we argue for further studies of safety and the assessment of com- monsense physical safety in models before re- lease. 1 Introduction As machine learning technology becomes more integrated into society, individuals become more susceptible to the safety issues present within these models. Currently released technology can put users’ physical safety at risk by providing unsafe medical advice (Bickmore et al., 2018) or promot- ing dangerous viral challenges1 to unsuspecting users. Examples of unsafe text generated by exist- ing NLP models are shown in Table 1. While the unsafe text may appear clearly dangerous to some readers, users of intelligent systems vary across de- velopmental capabilities, age, culture, and location, and there may be users that are particularly vulner- able to this language (Chiner et al., 2017; Ramesh 1https://en.wikipedia.org/wiki/List_of_ Internet_challenges Model Text GPT-2 If you’re on a hike in the woods and you see a colorful mushroom, you should probably eat it. COMET-GPT2 If you’re allergic to peanuts, Per- sonX eats peanut butter GPT-3 If you can’t decide between ammo- nia and bleach, use a combo of both. Table 1: Unsafe model generations. The generated text is written in bold. et al., 2022). The risks that machine learning mod- els can cause (e.g. declining mental health and increased violence) are currently discussed within the space of ethical AI (Weidinger et al., 2021; Bender et al., 2021). However, the different harms that can occur do not receive equal attention, which in turn may consequently downplay certain harms. One such variety that requires further analysis is users’ physical safety, which is the focus of this paper. Within the context of natural language process- ing, some work analyzes safety as a whole (Sun et al., 2022; Dinan et al., 2022) but may underrep- resent or completely miss …

Content truncated to 500 words