Situated Dialogue Learning through Procedural Environment Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8099 - 8116 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Situated Dialogue Learning through Procedural Environment Generation Prithviraj Ammanabrolu†‡ Renee Jia† †Georgia Institute of Technology ‡Allen Institute for AI raja@allenai.org Mark O. Riedl† Abstract We teach goal-driven agents to interactively act and speak in situated environments by training on generated curriculums. Our agents operate in LIGHT (Urbanek et al., 2019)—a large-scale crowd-sourced fantasy text adven- ture game wherein an agent perceives and in- teracts with the world through textual natu- ral language. Goals in this environment take the form of character-based quests, consist- ing of personas and motivations. We augment LIGHT by learning to procedurally generate additional novel textual worlds and quests to create a curriculum of steadily increasing diffi- culty for training agents to achieve such goals. In particular, we measure curriculum difficulty in terms of the rarity of the quest in the original training distribution—an easier environment is one that is more likely to have been found in the unaugmented dataset. An ablation study shows that this method of learning from the tail of a distribution results in significantly higher generalization abilities as measured by zero- shot performance on never-before-seen quests. 1 Introduction A key hypothesis in the pursuit towards creating goal-driven natural language-based agents posits that interactivity and environment grounding is critical for effective language learning (Barsalou, 2008; Bisk et al., 2020; Ammanabrolu and Riedl, 2021). Text games provide a platform on which to interactively train agents that can both act and speak in a situated manner—producing language that is both goal-driven and contextually relevant. Agents in text games operate—perceiving, acting in, and speaking to others in a world—entirely us- ing textual natural language. These games are struc- tured generally as sequential decision making prob- lems in the form of puzzles or quests that must be completed to advance in the game. As seen in Figure 1, we focus on creating agents in LIGHT (Urbanek et al., 2019), a large-scale Figure 1: The LIGHT questing environment presented as a 2 player game deployed in Messenger. crowdsourced fantasy text-adventure game, consist- ing of rich textual worlds—locations, objects, and characters with personas, and quests—motivations for each character. To complete these quests, an agent must: (1) maintain character via its persona; and (2) reason in a partially observable world about potential actions and utterances based on incom- plete descriptions of the locations, objects, and other characters. This requires several human like competencies such as commonsense reasoning, dy- namic natural language understanding, and operat- ing in combinatorially sized language-based state- action spaces. Although recent work has provided evidence showing that interactive language learn- ing via reinforcement learning (RL) in text games can be significantly more sample efficient than static supervised learning (Ammanabrolu et al., 8099 2021) when creating goal-driven natural language agents, their ability to robustly generalize to novel scenarios is limited. In sequential decision making problems in par- ticular, this generalization gap …

Content truncated to 500 words