SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6462 - 6481 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures Megan Ung Jing Xu Y-Lan Boureau Facebook AI Research {meganu, jingxu23,ylan}@fb.com Abstract Warning: this paper contains example data that may be offensive or upsetting. Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of- the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 8k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conver- sations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability. 1 Introduction Large neural generative dialogue models trained to mimic human English-language open-domain conversations have become engaging (Adiwardana et al., 2020; Roller et al., 2020b), but are still prone to uttering problematic language, e.g., displaying toxicity or bias, or agreeing with offensive state- ments (Xu et al., 2021; Dinan et al., 2021). Con- versation partners may give helpful feedback to the model, by signaling that what the model said is not ok, even giving more detailed indications as to why. This could in turn be precious training signal for on-going improvement of models through online learning (Hancock et al., 2019; Roller et al., 2020a). In particular, the boundaries of what constitutes ok Figure 1: Types of bot responses when responding to feedback about problematic inputs from the BAD dataset (Xu et al., 2021). Existing models (four bars on the left) respond by attacking or ignoring the feedback. Recovery models fine-tuned on the dataset of gracious responses to feedback proposed in this work apologize without defensiveness (two bars on the right). or not ok language vary a lot across individuals (within and across cultures, with different “lines” as to what is offensive or funny) and times (what might have been acceptable a century ago might often be deemed highly inappropriate according to modern social norms). Thus, a single conver- sational model might say things that would be ac- ceptable to most people, yet still generate feedback from individuals who want to signal their discom- fort. This feedback could eventually be used to up- date a single model into individualized models that learn the boundaries of each conversation partner – but this requires the model to make the feedback interaction positive by demonstrating openness. In- stead, current conversational models typically re- spond to feedback in a way that discourages the …

Content truncated to 500 words