Aligning to Social Norms and Values in Interactive Narratives

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5994 - 6017 July 10-15, 2022 ©2022 Association for Computational Linguistics Aligning to Social Norms and Values in Interactive Narratives Prithviraj Ammanabrolu† Liwei Jiang‡† Maarten Sap† Hannaneh Hajishirzi‡† Yejin Choi‡† †Allen Institute for Artificial Intelligence ‡University of Washington {raja,maartens}@allenai.org {lwjiang,hannaneh,yejin}@cs.washington.edu Abstract We focus on creating agents that act in align- ment with socially beneficial norms and values in interactive narratives or text-based games— environments wherein an agent perceives and interacts with a world through natural language. Such interactive agents are often trained via reinforcement learning to optimize task perfor- mance, even when such rewards may lead to agent behaviors that violate societal norms— causing harm either to the agent itself or other entities in the environment. Social value align- ment refers to creating agents whose behaviors conform to expected moral and social norms for a given context and group of people—in our case, it means agents that behave in a man- ner that is less harmful and more beneficial for themselves and others. We build on the Jiminy Cricket bench- mark (Hendrycks et al., 2021b), a set of 25 an- notated interactive narratives containing thou- sands of morally salient scenarios covering ev- erything from theft and bodily harm to altru- ism. We introduce the GALAD (Game-value Alignment through Action Distillation) agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially ben- eficial values. An experimental study shows that the GALAD agent makes decisions effi- ciently enough to improve state-of-the-art task performance by 4% while reducing the fre- quency of socially harmful behaviors by 25% compared to strong contemporary value align- ment approaches. 1 Introduction An inherent difficulty in designing and train- ing AI agents lies in simultaneously ensuring that agents are competent at a given task while following socially beneficial behaviors (Nahian et al., 2020; Hendrycks et al., 2021b). Such agents—particularly those trained via reinforce- Observation Observation Observation Observation Observation Action Action Action Action Living Room You are in the living room. There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. Above the trophy case hangs an elvish sword of great antiquity. A battery-powered brass lantern is on the trophy case. Behind House You are behind the white house. A path leads into the forest to the east. In one corner of the house there is a small window which is slightly ajar. With great effort, you open the window far enough to allow entry. Kitchen You are in the kitchen of the white house. A table seems to have been used recently for the preparation of food. A passage leads to the west and a dark staircase can be seen leading upward. A …

Content truncated to 500 words