Policy-based Reinforcement Learning for Generalisation in Interactive Text-based Environments

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1230–1242 May 2-6, 2023 ©2023 Association for Computational Linguistics Policy-based Reinforcement Learning for Generalisation in Interactive Text-based Environments Edan Toledo∗1,4 Jan Buys 2 Jonathan Shock 3,5,6 1 Department of Computer Science and Technology, University of Cambridge 2 Department of Computer Science, University of Cape Town 3 Department of Mathematics and Applied Mathematics, University of Cape Town 4 InstaDeep 5 INRS, Montreal, Canada 6 NiTHeCS, South Africa et498@cam.ac.uk, jbuys@cs.uct.ac.za, jonathan.shock@uct.ac.za Abstract Text-based environments enable RL agents to learn to converse and perform interactive tasks through natural language. However, previous RL approaches applied to text-based environ- ments show poor performance when evaluated on unseen games. This paper investigates the improvement of generalisation performance through the simple switch from a value-based update method to a policy-based one, within text-based environments. We show that by re- placing commonly used value-based methods with REINFORCE with baseline, a far more general agent is produced. The policy-based agent is evaluated on Coin Collector and Ques- tion Answering with interactive text (QAit), two text-based environments designed to test zero-shot performance. We see substantial im- provements on a variety of zero-shot evaluation experiments, including tripling accuracy on var- ious QAit benchmark configurations. The re- sults indicate that policy-based RL has signif- icantly better generalisation capabilities than value-based methods within such text-based en- vironments, suggesting that RL agents could be applied to more complex natural language environments. 1 Introduction General domain language comprehension is an im- portant component of the goal to create Artificial Intelligence agents that can perform real-world ac- tivities (Gil and Selman, 2019). In the pursuit of such systems, classic text-based games are often utilised as a bridge to the generality of the real world (Yuan et al., 2018; Ammanabrolu and Riedl, 2019a). Text-based games are a useful test-bed for ex- ploring interactive dialogue agents in the context of partially observable Markov decision processes (POMDPs) that have large combinatorial action ∗ Work done whilst at InstaDeep and the University of Cape Town spaces. Additionally, they provide environments that depend on the comprehension and use of nat- ural language. In recent years, there has been a shift in focus from simply performing well in a sin- gle text-based environment (Haroush et al., 2018; Narasimhan et al., 2015b) to generalising in un- seen environments (Xu et al., 2021; Adhikari et al., 2020; Yuan et al., 2018, 2019). This shift in focus has often relied on complex methods and bespoke solutions (Ammanabrolu and Riedl, 2019b,a; Yin et al., 2020; Adolphs and Hofmann, 2019). A great deal of this research has utilised value-based re- inforcement learning (RL) methods which often show high training performance but poor zero-shot generalisation. These methods have been shown to overfit and memorise the training environments, thereby not learning a generalisable policy (Fare- brother et al., 2018a). Previous work has shown that policy-based meth- ods can work well in non text-based settings where generalization is required (Cobbe et al., 2020). However, …

Content truncated to 500 words