Language-based General Action Template for Reinforcement Learning Agents

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2125–2139 August 1–6, 2021. ©2021 Association for Computational Linguistics 2125 Language-based General Action Template for Reinforcement Learning Agents Ryosuke Kohita IBM Research kohi@ibm.com Akifumi Wachi IBM Research akifumi.watchi@ibm.com Daiki Kimura IBM Research daiki@jp.ibm.com Subhajit Chaudhury IBM Research subhajit@jp.ibm.com Michiaki Tatsubori IBM Research mich@jp.ibm.com Asim Munawar IBM Resarch asim@ibm.com Abstract Prior knowledge plays a critical role in decision-making, and humans preserve such knowledge in the form of natural language (NL). To emulate real-world decision-making, artificial agents should incorporate such generic knowledge into their decision- making framework through NL. However, since policy learning with NL-based action representation is intractable due to NL’s combinatorial complexity, previous studies have limited agents’ expressive power to only a specific environment, which sacrificed the generalization ability to other environments. This paper proposes a new environment- agnostic action framework, the language- based general action template (L-GAT). We design action templates on the basis of general semantic schemes (FrameNet, VerbNet, and WordNet), facilitating the agent in finding a plausible action in a given state by using prior knowledge while covering broader types of actions in a general manner. Our experiment using 18 text-based games showed that our proposed L-GAT agent which uses the same actions across games, achieved a performance competitive with agents that rely on game- specific actions. We have published the code at https://github.com/kohilin/lgat. 1 Introduction The incorporation of natural language processing (NLP) and reinforcement learning (RL) is an important research field for using knowledge, represented in the form of language, in the decision-making of artificial agents (Luketina et al., 2019). One critical topic is the capability to describe an agent’s actions with natural language (NL) (Narasimhan et al., 2015; Yuan et al., 2019). An agent with such a capability can estimate the plausibility of actions on the basis of prior knowledge accessible through GO GIVE EAT … FrameNet VerbNet WordNet Environment- dependent Weak prior knowledge Go west Enter house Drink tea Go kitchen Chop meat Eat potato … … Language-based General Action Template Environment- agnostic Strong prior knowledge Common Natural Language-based Action Set (Envs) Figure 1: Common action templates and L-GAT. language (Narasimhan et al., 2018). Suppose that an agent receives a request, “give me some water”; a common sense idea like “water exists in the kitchen” will definitely help the agent determine the right direction to go in. If actions are represented in language, for example, GO TO KITCHEN, we can straightforwardly connect knowledge to actions by referring to language resources (Fulda et al., 2017). NL is useful for accessing knowledge to achieve plausible decision- making, and this language capability would be fundamental in developing intelligent agents. However, NL is complicated for current RL agents to acquire due to its high expressive power (Hausknecht et al., 2020). The rich vocabulary and complex grammar of NL result in a huge action space that is intractable for existing RL algorithms. Although we can train an agent by restricting the expressive power to a specific environment (Narasimhan et …

Content truncated to 500 words