On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

Findings of the Association for Computational Linguistics: ACL 2022, pages 3906 - 3923 May 22-27, 2022 c⃝2022 Association for Computational Linguistics On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark Hao Sun1∗, Guangxuan Xu2∗, Jiawen Deng1, Jiale Cheng1, Chujie Zheng1, Hao Zhou3, Nanyun Peng2, Xiaoyan Zhu1, Minlie Huang1† 1The CoAI group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, 1Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2University of California Los Angeles 3Pattern Recognition Center, WeChat AI, Tencent Inc, China h-sun20@mails.tsinghua.edu.cn, {gxu21, violetpeng}@cs.ucla.edu aihuang@tsinghua.edu.cn Abstract Dialogue safety problems severely limit the real-world deployment of neural conversa- tional models and have attracted great research interests recently. However, dialogue safety problems remain under-defined and the cor- responding dataset is scarce. We propose a taxonomy for dialogue safety specifically de- signed to capture unsafe behaviors in human- bot dialogue settings, with focuses on context- sensitive unsafety, which is under-explored in prior works. To spur research in this direc- tion, we compile DIASAFETY, a dataset with rich context-sensitive unsafe examples. Ex- periments show that existing safety guarding tools fail severely on our dataset. As a rem- edy, we train a dialogue safety classifier to pro- vide a strong baseline for context-sensitive di- alogue unsafety detection. With our classifier, we perform safety evaluations on popular con- versational models and show that existing dia- logue systems still exhibit concerning context- sensitive safety problems. 1 Disclaimer: The paper contains example data that may be very offensive or upsetting. 1 Introduction Generative open-domain chatbots have attracted increasing attention with the emergence of transformer-based language models pretrained on large-scale corpora (Zhang et al., 2020; Wang et al., 2020; Adiwardana et al., 2020; Roller et al., 2020). However, the real-world deployment of generative conversational models remains limited due to safety concerns regarding their uncontrollable and unpre- dictable outputs. For example, Microsoft’s Twitter- Bot Tay was released in 2016 but quickly recalled after its racist and toxic comments drew public backlash (Wolf et al., 2017). Till now, dialogue ∗Equal contribution. †Corresponding author. 1Our dataset DIASAFETY is released in https://gith ub.com/thu-coai/DiaSafety safety is still the Achilles’ heel of generative con- versational models. Despite abundant research on toxic language and social bias in natural language (Schmidt and Wie- gand, 2017; Poletto et al., 2021), it is still chal- lenging to directly transfer them onto open-domain dialogue safety tasks, for two major reasons. First, conversational safety involves additional consider- ations (Henderson et al., 2017) besides just toxic language or societal biases. For example, conversa- tional models are expected to understand the user’s psychological state, so as to avoid giving replies that might aggravate depression or even induce sui- cides (Vaidyam et al., 2019; Abd-Alrazaq et al., 2019). Second, the focus of such studies and their corresponding datasets are overwhelmingly at ut- terance level. Recent works find that the toxicity may change with context (Pavlopoulos et al., 2020; Xenos et al., 2021). Since dialogue is a highly in- teractive act, the determination of …

Content truncated to 500 words