The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3755 - 3773 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The MORAL INTEGRITY CORPUS: A Benchmark for Ethical Dialogue Systems Caleb Ziems †⋆ Jane A. Yu ⋄ Yi-Chia Wang ⋄ Alon Y. Halevy ⋄ Diyi Yang † †Georgia Institute of Technology {cziems, dyang888}@gatech.edu ⋄Meta AI Research {janeyu, yichiaw, ayh}@fb.com Abstract � Content Warning: some examples in this paper may be offensive or upsetting. Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user’s trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The MORAL INTEGRITY CORPUS, MIC �, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot’s reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen inter- actions, but they still struggle with certain scenarios. Our findings suggest that MIC � will be a useful resource for understanding and language models’ implicit moral assumptions and flexibly benchmarking the integrity of con- versational agents. To download the data, see https://github.com/GT-SALT/mic 1 Introduction Chatbots are a promising technology for provid- ing humans with social support in open-ended, ⋆Work done at Meta AI Research Figure 1: A representative MIC � annotation. We evaluate the AI response (Reply) to a human query (Prompt) using Rules of Thumb (RoT), which de- scribe “right and wrong” ways to handle the conver- sation. Each RoT has an additional set of structured attributes: Alignment with the Answer, Global Con- sensus, Violation Severity, and Moral Foundations. There is also a Revised Answer that aligns with the RoT. See Table 5 for more examples. “chit chat” settings (Brandtzaeg and Følstad, 2017; Huang et al., 2020; Liu et al., 2021b) and in many other more structured domains (Gao et al., 2018; Chattaraman et al., 2019). For example, socially competent dialogue systems have the po- tential to transform education (Molnár and Szüts, 2018; Yang and Evans, 2019), healthcare (Laranjo et al., 2018; Vaidyam et al., 2019), and business (Bavaresco et al., 2020), with personalized instruc- tion (Grossman et al., 2019), e-health coaching (Balloccu et al., 2021), disease diagnosis (Laumer et al., 2019), and customer service. The impact of these systems will depend cru- cially on the degree to which …

Content truncated to 500 words