Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9379 - 9393 December 7-11, 2022 ©2022 Association for Computational Linguistics Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence Chris Callison-Burch∗ University of Pennsylvania ccb@upenn.edu Gaurav Singh Tomar∗ Google Research gtomar@google.com Lara J. Martin University of Pennsylvania Daphne Ippolito University of Pennsylvania Google Research Suma Bailis Google Research David Reitter Google Research Abstract AI researchers have posited Dungeons and Dragons (D&D) as a challenge problem to test systems on various language-related capabil- ities. In this paper, we frame D&D specif- ically as a dialogue system challenge, where the tasks are to both generate the next conver- sational turn in the game and predict the state of the game given the dialogue history. We cre- ate a gameplay dataset consisting of nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns, 500,000 dice rolls, and 58 mil- lion words. We automatically annotate the data with partial state information about the game play. We train a large language model (LM) to generate the next game turn, condi- tioning it on different information. The LM can respond as a particular character or as the player who runs the game—i.e., the Dungeon Master (DM). It is trained to produce dialogue that is either in-character (roleplaying in the fictional world) or out-of-character (discussing rules or strategy). We perform a human evalu- ation to determine what factors make the gen- erated output plausible and interesting. We fur- ther perform an automatic evaluation to deter- mine how well the model can predict the game state given the history and examine how well tracking the game state improves its ability to produce plausible conversational output. 1 Introduction Artificial Intelligence has a long and rich history of using games as challenge problems that lead to ad- vances in the field. In many cases, AI game-playing systems have gone on to rival human champions of the game. Dungeons and Dragons has been identi- fied as an appropriate challenge for the next stage of artificial intelligence (Ellis and Hendler, 2017; Louis and Sutton, 2018; Martin et al., 2018b). Ellis and Hendler (2017) proposed open-ended creative games like D&D as the next challenge for AI af- ter the human-level successes of AI at Chess and ∗*Denotes equal contribution Go, which are zero-sum, deterministic, sequential two-player games with perfect information. Louis and Sutton (2018) understood the importance of narrative in natural language processing (NLP) and generation (NLG). In particular, they saw how co- operative story generation between humans already exists in these games and can be used for auto- mated generation. Martin et al. (2018b) outlined some of the specific challenges D&D presents to the NLP community; such as a state of the game world distributed across the Dungeon Master (DM) and other players or dealing with the intrinsic re- wards players get from taking certain actions that would not necessarily provide them with points in the game. D&D involves multiple players who roleplay characters in …

Content truncated to 500 words