PLM-based World Models for Text-based Games

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1324 - 1341 December 7-11, 2022 ©2022 Association for Computational Linguistics PLM-based World Models for Text-based Games Minsoo Kim♠ YeonJoon Jung♣ Dohyeon Lee♢ Seung-won Hwang∗♢ ♠Interdisciplinary Program in Artificial Intelligence, Seoul National University ♢Department of Computer Science and Engineering, Seoul National University ♣Department of Artificial Intelligence, Yonsei University {minsoo9574, waylight3, seungwonh}@snu.ac.kr {theaitetus}@yonsei.ac.kr Abstract World models have improved the ability of rein- forcement learning agents to operate in a sam- ple efficient manner, by being trained to predict plausible changes in the underlying environ- ment. As the core tasks of world models are fu- ture prediction and commonsense understand- ing, our claim is that pre-trained language mod- els (PLMs) already provide a strong base upon which to build world models. Worldformer is a recently proposed world model for text-based game environments, based only partially on PLM and transformers. Our distinction is to fully leverage PLMs as actionable world mod- els in text-based game environments, by refor- mulating generation as constrained decoding which decomposes actions into verb templates and objects. We show that our model improves future valid action prediction and graph change prediction.1 Additionally, we show that our model better reflects commonsense than stan- dard PLM. 1 Introduction In model-based reinforcement learning, world mod- els (Ha and Schmidhuber, 2018), being trained to predict plausible changes in the underlying environ- ment, has helped agents to quickly identify sensible or high-value actions. As a result, agents equipped with world models have outperformed state-of-the art model-free algorithms in Atari games, while drastically improving sample efficiency (Kaiser et al., 2019; Hafner et al., 2020). In this work, we focus on world modeling in text- based game (TBG) environments, in which play- ers and agents must perceive and interact with the world entirely through textual natural language. As such, they present several unique challenges (Côté et al., 2018; Hausknecht et al., 2019; Ammanabrolu ∗Corresponding author. 1Code and data are available at: https://github.com/ mnskim/awm-bart Figure 1: Model architecture of AWM-BART, illus- trating the dual world modeling tasks of JerichoWorld. BART encoder-decoder denoted in blue learns the future valid actions prediction task, while the one denoted in orange simultaneously learns the graph difference pre- diction task. The light yellow blocks denote generation constrained by a template, bringing generation closer to the target controlled sublanguage of the text-based game’s parser. and Riedl, 2021b): Aside from language under- standing itself, TBGs require dealing with an expo- nential action space: For instance, the action space size for Zork1, combining five words from a vocab- ulary of 697 words, is O(1014). TBGs also require rich and accurate knowledge representations of lo- cations and objects, in order to facilitate navigation and interaction. Finally, solving TBGs requires commonsense reasoning, including understanding object affordance, and the causal ramifications of actions. The first dataset and model for learn- ing world models in TBGs were proposed by Jeri- choWorld (Ammanabrolu and Riedl, 2021b), and Worldformer (Ammanabrolu and Riedl, 2021a), re- spectively. Intuitively, large pre-trained …

Content truncated to 500 words