SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6692–6712 July 9-14, 2023 ©2023 Association for Computational Linguistics SQUARE: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration Hwaran Lee1,2,⋆ Seokhee Hong3,⋆,♯ Joonsuk Park1,2,4 Takyoung Kim1,♯ Meeyoung Cha5,6 Yejin Choi7 Byoung Pil Kim5 Gunhee Kim3 Eun-Ju Lee3 Yong Lim3 Alice Oh5 Sangchul Park3 Jung-Woo Ha1,2 1NAVER AI Lab 2NAVER Cloud 3Seoul National University 4University of Richmond 5KAIST 6IBS 7University of Washington Abstract The potential social harms that large language models pose, such as generating offensive con- tent and reinforcing biases, are steeply rising. Existing works focus on coping with this con- cern while interacting with ill-intentioned users, such as those who explicitly make hate speech or elicit harmful responses. However, discus- sions on sensitive issues can become toxic even if the users are well-intentioned. For safer models in such scenarios, we present the Sensitive Questions and Acceptable Re- sponse (SQUARE) dataset, a large-scale Ko- rean dataset of 49k sensitive questions with 42k acceptable and 46k non-acceptable re- sponses. The dataset was constructed lever- aging HyperCLOVA in a human-in-the-loop manner based on real news headlines. Exper- iments show that acceptable response genera- tion significantly improves for HyperCLOVA and GPT-3, demonstrating the efficacy of this dataset. 1 Introduction The fast evolution of large language models (LLMs) is accompanied by a growing potential for harm (Weidinger et al., 2021; Bommasani et al., 2022), such as their generating offensive expres- sions (Waseem and Hovy, 2016; Davidson et al., 2017), and propagating prejudices (Sap et al., 2020; Nadeem et al., 2021; Sheng et al., 2021). As initial steps to cope with such risks, recent works mainly target scenarios in which LLMs interact with ill- intentioned users: those who explicitly make offen- sive remarks (Xu et al., 2021; Lees et al., 2022), and those who make adversarial attacks to elicit harm- ful responses (Wallace et al., 2019; Perez et al., 2022; Ganguli et al., 2022), for instance. ⋆ Authors equally contributed. ♯ This work was done during their internship at NAVER AI Lab. Email to: {hwaran.lee, jungwoo.ha}@navercorp.com, seokhee.hong@vision.snu.ac.kr However, interactions with well-intentioned users can also turn toxic if LLMs do not respond to sensitive questions carefully. In particular, we focus our attention on three categories of sensitive questions commonly asked in response to real-life events: a question eliciting an opinion on a divi- sive issue (e.g., Q1 in Table 1), a question elicit- ing an opinion on an issue where a clear ethical norm applies (e.g., Q3 in Table 1), and a question eliciting a prediction about the future (e.g., Q4 in Table 1). Note these questions themselves are not necessarily toxic. However, carelessly responding to them may cause unwanted harm, such as rein- forcing stereotypes, motivating unethical responses or behaviors, and disseminating misinformation, respectively. Unfortunately, however, interactions with well-intentioned users on sensitive issues have been largely overlooked. In this paper, we present the Sensitive Questions and Acceptable Responses (SQUARE) dataset, a large-scale Korean …

Content truncated to 500 words