How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 807–833 June 6–11, 2021. ©2021 Association for Computational Linguistics 807 How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds Prithviraj Ammanabrolu† Jack Urbanek‡ Margaret Li‡ Arthur Szlam‡ Tim Rocktäschel‡ Jason Weston‡ †Georgia Institute of Technology ‡Facebook AI Research raj.ammanabrolu@gatech.edu Abstract We seek to create agents that both act and com- municate with other agents in pursuit of a goal. Towards this end, we extend LIGHT (Urbanek et al., 2019)—a large-scale crowd-sourced fan- tasy text-game—with a dataset of “quests”.1. These contain natural language motivations paired with in-game goals and human demon- strations; completing a quest might require di- alogue or actions (or both). We introduce a re- inforcement learning system that (1) incorpo- rates large-scale language modeling-based and commonsense reasoning-based pre-training to imbue the agent with relevant priors; and (2) leverages a factorized action space of action commands and dialogue, balancing between the two. We conduct zero-shot evaluations using held-out human expert demonstrations, showing that our agents are able to act consis- tently and talk naturally with respect to their motivations. 1 Introduction There has been a recent improvement in the quality of natural language processing (NLP) and gener- ation (NLG) by machine learning (ML) (Vaswani et al., 2017; Devlin et al., 2018); and in parallel, improvement to goal-oriented ML driven agents in the context of games (Vinyals et al., 2019; Schrit- twieser et al., 2019). However, agents that can com- municate with humans (and other agents) through natural language in pursuit of their goals are still primitive. One possible reason for this is that many datasets and tasks used for NLP are static, not supporting interaction and language ground- ing (Brooks, 1991; Feldman and Narayanan, 2004; Barsalou, 2008; Mikolov et al., 2016; Gauthier and Mordatch, 2016; Lake et al., 2017). Text-based games—where players see, act upon, and com- municate within a dynamic world using natural 1Data can be found here https://parl.ai/ projects/light/ language—provide a platform on which to develop such goal-driven agents. LIGHT (Urbanek et al., 2019), a large-scale crowdsourced fantasy text-adventure game, consist- ing of a set of locations, characters, and objectsa possesses rich textual worlds, but without any no- tion of goals to train goal-driven agents. We present a dataset of quests for LIGHT and demonstrations of humans playing these quests (as seen in Fig- ures 2 and 3), providing natural language descrip- tions in varying levels of abstraction of motivations for a given character in a particular setting. To complete these quests, an agent must reason about potential actions and utterances based on in- complete descriptions of the locations, objects, and other characters. When a human is placed in a fantasy setting such as LIGHT, they already know that kings are royalty and must be treated respect- fully, swords are weapons, etc.—commonsense knowledge that a learning agent must acquire to en- sure successful interactions. To equip agents with relevant priors in …

Content truncated to 500 words