Self-imitation Learning for Action Generation in Text-based Games

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 703–726 May 2-6, 2023 ©2023 Association for Computational Linguistics Self-imitation Learning for Action Generation in Text-based Games Zijing Shi1, Yunqiu Xu1, Meng Fang2, Ling Chen1 1 University of Technology Sydney, NSW, Australia zijing.shi@student.uts.edu.au, {yunqiu.xu,ling.chen}@uts.edu.au 2 University of Liverpool, Liverpool, UK Meng.Fang@liverpool.ac.uk Abstract In this work, we study reinforcement learning (RL) in solving text-based games. We address the challenge of combinatorial action space, by proposing a confidence-based self-imitation model to generate action candidates for the RL agent. Firstly, we leverage the self-imitation learning to rank and exploit past valuable tra- jectories to adapt a pre-trained language model (LM) towards a target game. Then, we devise a confidence-based strategy to measure the LM’s confidence with respect to a state, thus adap- tively pruning the generated actions to yield a more compact set of action candidates. In multiple challenging games, our model demon- strates promising performance in comparison to the baselines. 1 Introduction Text-based games are situated systems where the game agents observe textual descriptions, and gen- erate textual commands to interact with the en- vironment. These games have proven to be suit- able test-beds for studying various natural language processing (NLP) tasks, such as question answer- ing (Yuan et al., 2019; Xu et al., 2022), dialogue systems (Ammanabrolu et al., 2022), situated lan- guage learning (Shridhar et al., 2020) and common- sense reasoning (Murugesan et al., 2021; Ryu et al., 2022). Recent years have witnessed the thriveness of designing reinforcement learning (RL) agents in solving these games (Narasimhan et al., 2015; Hausknecht et al., 2020; Ammanabrolu and Riedl, 2019; Xu et al., 2020b), while the combinatorial action space remains as a challenging issue, pre- venting RL agents from being deployed in real world applications. In general, text-based games accept free-form actions, resulting in a large combinatorial action space. Fig. 1 shows a raw excerpt from the classic game “Zork1”. A 4-word action has to be selected from |V|4 candidates, where V denotes the vocab- ulary set (Xu et al., 2020a). Given that only 130 Figure 1: Excerpt from the game “Zork1”. With a vocabulary size of 697, there are around 6974 ≈ 200 billion potential 4-word actions in the game. actions are required to solve this game, the agent wastes both training data and time in attempting irrelevant actions (Dulac-Arnold et al., 2015). To handle the combinatorial action space, early efforts either heavily rely on hand-crafted rules, or sim- ply assume the availability of the action candidate set. For example, some works consider a set of currently admissible actions (He et al., 2016), or a template-based action space (Hausknecht et al., 2020). Alternatively, some other works allevi- ated this challenge by filtering inadmissible actions through methods such as action affordance (Jain et al., 2020), bandit-based elimination (Zahavy et al., 2018) and rule-based scoring (Ammanabrolu and Riedl, 2019). 703 In handling the combinatorial action space for text-based games, recent pre-trained language mod- els (PLMs) (Devlin et al., …

Content truncated to 500 words