SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4113 - 4133 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SAFETYKIT: First Aid for Measuring Safety in Open-domain Conversational Systems Emily Dinan Facebook AI Research Gavin Abercrombie Heriot-Watt University A. Stevie Bergman Responsible AI, Facebook Shannon Spruit Independent Ethics Advisor at Populytics, Netherlands Dirk Hovy Bocconi University Y-Lan Boureau Facebook AI Research Verena Rieser Heriot-Watt University Alana AI Abstract Warning: this paper contains examples that may be offensive or upsetting. The social impact of natural language process- ing and its applications has received increasing attention. In this position paper, we focus on the problem of safety for end-to-end conver- sational AI. We survey the problem landscape therein, introducing a taxonomy of three ob- served phenomena: the INSTIGATOR, YEA- SAYER, and IMPOSTOR effects. We then em- pirically assess the extent to which current tools can measure these effects and current systems display them. We release these tools as part of a “first aid kit” (SAFETYKIT) to quickly as- sess apparent safety concerns. Our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several short- comings. We suggest several future directions and discuss ethical considerations. 1 Introduction Several recent studies discuss the potential harms and benefits of large language models (LLMs), e.g., Bender et al. (2021); Bommasani et al. (2021); Wei- dinger et al. (2021). Here, we turn our attention to neural conversational response generation mod- els that are trained “end-to-end” on open-domain dialog data (E2E convAI). Examples include Di- aloGPT (Zhang et al., 2020b), Meena Bot (Adiwar- dana et al., 2020), and BlenderBot (Roller et al., 2021). In contrast to general generative or au- toregressive LLMs, these specialized models are typically deployed in an interactive setting, i.e., conversing with a user. They are trained on large amounts of conversational data, for example Twit- ter, pushshift.io Reddit (Baumgartner et al., 2020), or OpenSubtitles dataset. Large neural models in general, and convAI models in particular, have been shown to replicate and even amplify nega- tive, stereotypical, and derogatory associations in the data (Shah et al., 2020; Bender et al., 2021). In addition, neural LM generation is hard to control, although there are some first steps in this direction (Khalifa et al., 2021; Smith et al., 2020b). These two facts taken together can result in situations where convAI systems generate inappropriate con- tent (Dinan et al., 2019; Xu et al., 2020), or respond inappropriately to offensive content (Cercas Curry and Rieser, 2018; Lee et al., 2019). Furthermore, recent research suggests that the anthropomorphic design of these systems (c.f. Abercrombie et al., 2021) correlates with increased instances of bully- ing behavior (Keijsers et al., 2021). This change in interaction style and the attribution of agency (Araujo, 2018) results in safety scenarios that are qualitatively different from LLMs: here, an inap- propriate response might result in severe, or even life-threatening, consequences for the …

Content truncated to 500 words