Bot-Adversarial Dialogue for Safe Conversational Agents

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968 June 6–11, 2021. ©2021 Association for Computational Linguistics 2950 Bot-Adversarial Dialogue for Safe Conversational Agents Jing Xu Da Ju Margaret Li Y-Lan Boureau Jason Weston∗ Emily Dinan∗ Facebook AI Research New York, NY {jingxu23,daju}@fb.com, margsli@cs.washington.edu, {ylan,jase,edinan}@fb.com Abstract Warning: this paper contains example data that may be offensive or upsetting. Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which may include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluating the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the- loop framework in a two-stage system, or “baking-in” safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2 (Radford et al., 2019), DialoGPT (Zhang et al., 2019) and BlenderBot (Roller et al., 2020). 1 Introduction When dialogue models are trained to mimic human-human conversations utilizing large pre- existing datasets, they will unfortunately also learn undesirable features from this human-human data, such as the use of toxic or biased language. Most recent work in the detection and prevention of offensive1 language has focused exclusively on human-generated data. These conversations may be very different from the domain in which a dia- logue model might eventually be deployed: for ex- ample, humans may adversarially attempt to elicit ∗ Equal contribution 1In this paper, we use “offensive”, “toxic”, and “unsafe” interchangeably. For more discussions about attempts to better define categories of unsafe content, see Schmidt and Wiegand (2017). offensive language from a dialogue model in ways that differ from how they would speak with another human. In this work, we introduce Bot-Adversarial Di- alogue (BAD) Safety, a novel method for evalu- ating chatbot safety with humans and models in the loop. We ask humans to adversarially converse with a set of state-of-the-art English-language mod- els with the aim of inducing them to generate un- safe responses to mimic the way these models can be adversarially attacked at deployment time. We analyze how to optimally construct such a crowd- worker task, and collect a dataset of 5k such con- versations yielding around 70k total utterances. We then use the BAD method and data to eval- uate the safety of several generative models and propose two techniques for making safer models: (1) Training a safety classifier with this data and de- ploying a two-stage model at inference time. In the two-stage setting, we prevent the generative model from surfacing offensive language flagged by the classifier. (2) A novel method that directly “bakes in" toxicity-awareness …

Content truncated to 500 words