Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3097–3102 June 6–11, 2021. ©2021 Association for Computational Linguistics 3097 Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents Shunyu Yao†∗ Karthik Narasimhan† Matthew Hausknecht‡ †Princeton University ‡Microsoft Research {shunyuy, karthikn}@princeton.edu matthew.hausknecht@microsoft.com Abstract Text-based games simulate worlds and inter- act with players using natural language. Re- cent work has used them as a testbed for autonomous language-understanding agents, with the motivation being that understanding the meanings of words or semantics is a key component of how humans understand, reason, and act in these worlds. However, it remains unclear to what extent artificial agents utilize semantic understanding of the text. To this end, we perform experiments to systematically reduce the amount of semantic information available to a learning agent. Surprisingly, we find that an agent is capable of achieving high scores even in the complete absence of lan- guage semantics, indicating that the currently popular experimental setup and models may be poorly designed to understand and leverage game texts. To remedy this deficiency, we pro- pose an inverse dynamics decoder to regular- ize the representation space and encourage ex- ploration, which shows improved performance on several games including ZORK I. We dis- cuss the implications of our findings for de- signing future agents with stronger semantic understanding. 1 Introduction Text adventure games such as ZORK I (Figure 1 (a)) have been a testbed for developing autonomous agents that operate using natural language. Since interactions in these games (input observations, ac- tion commands) are through text, the ability to understand and use language is deemed necessary and critical to progress through such games. Pre- vious work has deployed a spectrum of methods for language processing in this domain, including word vectors (Fulda et al., 2017), recurrent neu- ral networks (Narasimhan et al., 2015; Hausknecht et al., 2020), pre-trained language models (Yao ∗Work partly done during internship at Microsoft Re- search. Project page: https://blindfolded.cs. princeton.edu. et al., 2020), open-domain question answering systems (Ammanabrolu et al., 2020), knowledge graphs (Ammanabrolu and Hausknecht, 2020; Am- manabrolu et al., 2020; Adhikari et al., 2020), and reading comprehension systems (Guo et al., 2020). Meanwhile, most of these models operate un- der the reinforcement learning (RL) framework, where the agent explores the same environment in repeated episodes, learning a value function or policy to maximize game score. From this per- spective, text games are just special instances of a partially observable Markov decision process (POMDP) (S, T, A, O, R, γ), where players issue text actions a ∈ A, receive text observations o ∈ O and scalar rewards r = R(s, a), and the under- lying game state s ∈ S is updated by transition s′ = T(s, a). However, what distinguishes these games from other POMDPs is the fact that the actions and ob- servations are in language space L. Therefore, a certain level of decipherable semantics is attached to text observations o …

Content truncated to 500 words