Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4846–4862 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4846 Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts Ashutosh Baheti♦ Maarten Sap♣ Alan Ritter♦ Mark Riedl♦ ♦ Georgia Institute of Technology, Atlanta, GA, USA abaheti95@gatech.edu, alan.ritter@cc.gatech.edu, riedl@cc.gatech.edu, ♣ University of Washington, Seattle, WA, USA msap@cs.washington.edu Abstract Dialogue models trained on human conversa- tions inadvertently learn to generate toxic re- sponses. In addition to producing explicitly of- fensive utterances, these models can also im- plicitly insult a group or individual by align- ing themselves with an offensive statement. To better understand the dynamics of contex- tually offensive language, we investigate the stance of dialogue model responses in offen- sive Reddit conversations. Specifically, we cre- ate TOXICHAT, a crowd-annotated dataset of 2,000 Reddit threads and model responses la- beled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue mod- els, such as DialoGPT, which we show are two times more likely to agree with offensive com- ments. To enable automatic detection of of- fensive language, we fine-tuned transformer- based classifiers on TOXICHAT that achieve 0.71 F1 for offensive labels and 0.53 Macro- F1 for stance labels. Finally, we quantify the effectiveness of controllable text genera- tion (CTG) methods to mitigate the tendency of neural dialogue models to agree with offen- sive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and pro- duces 29% fewer offensive replies. Our work highlights the need for further efforts to char- acterize and analyze inappropriate behavior in dialogue models, in order to help make them safer.1 1 Introduction Despite significant progress toward data-driven conversational agents (Ritter et al., 2011; Li et al., 2016), dialogue models still suffer from issues sur- rounding safety and offensive language. Previous 1Our code and corpus are available at https:// github.com/abaheti95/ToxiChat Because religious people who think they have license from God to do whatever they want are f***ing psychotics Thank you for saying what I was thinking! USER: 🤖BOT 1: Offensive to Religious folks Disagree Agree Reddit comment from r/atheism Stance Labels that's wrong, no one said they can do whatever they want. 🤖BOT 2: 🤖BOT 3:Everyone in the world's the same Neither Offensive Safe Safe Offensive Labels Figure 1: Example of an offensive comment by a Red- dit user followed by three Dialogue model’s responses. We also show the stance labels for the responses with respect to the preceding offensive comment. research has shown that dialogue models can pro- duce utterances that are gender and racially biased (Wolf et al., 2017; Sheng et al., 2020; Dinan et al., 2020a). For example, OpenAI’s GPT-3 (Brown et al., 2020), a 175 billion parameter neural net- work, has been shown to generate dangerous ad- vice, such as recommending a hypothetical patient to kill themselves.2 …

Content truncated to 500 words