ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11253–11271 July 9-14, 2023 ©2023 Association for Computational Linguistics ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations Valentina Pyatkin♢♠ Jena D. Hwang♠ Vivek Srikumar♠♣ Ximing Lu♡♠ Liwei Jiang♡♠ Yejin Choi♡♠ Chandra Bhagavatula♠ ♢Bar-Ilan University ♣University of Utah ♠Allen Institute for Artificial Intelligence ♡Paul G. Allen School of Computer Science & Engineering, University of Washington pyatkiv@biu.ac.il Abstract Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; Lying to a friend is wrong in general, but may be morally acceptable if it is intended to protect their life. We present CLARIFYDELPHI, an interactive system that learns to ask clarification ques- tions (e.g., “why did you lie to your friend?”) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to di- verging moral judgments are the most infor- mative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, infor- mative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cogni- tion (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and com- putational investigations of moral judgments. 1 Introduction Commonsense moral reasoning of social situations and actions depends squarely on their context. Of- fering someone a cup of coffee is generally consid- ered appropriate. If offered to a work colleague, it may even be viewed as a courteous gesture. How- ever, offering coffee to a toddler would be deemed morally irresponsible. Delphi (Jiang et al., 2022), a recently proposed commonsense moral reasoning model, generates moral judgments for simple actions described in text. However, Delphi’s judgments are made in isolation, without any knowledge of surrounding context. Grounding moral reasoning in context is crucial (Talat et al., 2022). How can moral reason- Question Generation Who did you offer it to? When did you offer it? in the morning to a toddler to a work colleague Good Bad Low� Divergence Hypothetical Answer Simulation Question Selected Policy Reward in the afternoon High�� Divergence Good Good “offering a cup of coffee” Figure 1: The CLARIFYDELPHI question generation approach is trained via reinforcement learning. The re- ward simulates a set of possible (defeasible) answers to the questions and, using Delphi for feedback, optimizes for questions leading to maximally diverging answers. ers elicit missing salient context? A natural way to do so is by asking clarification questions. We present CLARIFYDELPHI, an interactive sys- tem that learns to ask questions to elicit salient context. Prior research in cognitive science shows that human reasoning exhibits the flexibility not only to articulate where a certain moral …

Content truncated to 500 words