A Survey of Text Games for Reinforcement Learning Informed by Natural Language

A Survey of Text Games for Reinforcement Learning Informed by Natural Language Philip Osborne Department of Computer Science University of Manchester United Kingdom philiposbornedata@gmail.com Heido N˜omm Department of Computer Science University of Manchester United Kingdom heidonomm@gmail.com Andr´e Freitas Department of Computer Science University of Manchester United Kingdom andre.freitas@manchester.ac.uk Abstract Reinforcement Learning has shown success in a number of complex virtual environments. However, many challenges still exist towards solving problems with natural language as a core component. Interactive Fiction Games (or Text Games) are one such problem type that offer a set of safe, partially observable environments where natural language is re- quired as part of the Reinforcement Learning solution. Therefore, this survey’s aim is to assist in the development of new Text Game problem settings and solutions for Reinforce- ment Learning informed by natural language. Specifically, this survey: 1) introduces the chal- lenges in Text Game Reinforcement Learn- ing problems, 2) outlines the generation tools for rendering Text Games and the subsequent environments generated, and 3) compares the agent architectures currently applied to provide a systematic review of benchmark methodolo- gies and opportunities for future researchers. 1 Introduction Language is often used by humans to abstract, transfer, and communicate knowledge of their de- cision making when completing complex tasks. However, traditional Reinforcement Learning (RL) methods, such as the prominent neural agents introduced by Mnih et al. (2013) and Silver et al. (2016), are limited to single task environments defined and solved without any language. Luketina et al. (2019) specified that further studies are required for improving solutions on problems that necessitate the use of language. The authors also note that language may improve solutions on problems that are traditionally solved without language. This includes the use of neural agents pre-trained on natural language corpora transferring syntactic and semantic information to future tasks. An example is the use of the similarities between the tasks of chopping and cutting a carrot in CookingWorld (Trischler et al., 2019)—that is, completing one should allow you to transfer the decision making to the other. To aid in developing solutions on these prob- lems, we pose Text Games as a testing environ- ment as they simulate complex natural language problems in controllable settings. In other words, researchers can generate the Text Games with limitations to evaluate any number of the specific challenges given by Dulac-Arnold et al. (2019) and Luketina et al. (2019). Detailed descriptions of the possible controls (defined as ‘handicaps’) are provided in Section 2.4. Likewise, the challenges with currently posed solutions are provided in Section 2.3 and are summarized by the following: • Partial observability - observed information is only a limited view of the underlying truth (shown in Figure 1). • Large state space - balancing exploration to new states against exploiting known paths. • Large and sparse action space - language increases the number of actions as there are multiple ways to describe the same input. • Long-term credit assignment - learning which actions are important when reward 873 Transactions of …

Content truncated to 500 words