Semi-Supervised Policy Initialization for Playing Games with Language Hints

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3112–3116 June 6–11, 2021. ©2021 Association for Computational Linguistics 3112 Semi-Supervised Policy Initialization for Playing Games with Language Hints Tsu-Jui Fu and William Yang Wang UC Santa Barbara {tsu-juifu, william}@cs.ucsb.edu Abstract Using natural language as a hint can supply an additional reward for playing sparse-reward games. Achieving a goal should involve sev- eral different hints, while the given hints are usually incomplete. Those unmentioned latent hints still rely on the sparse reward signal, and make the learning process difficult. In this pa- per, we propose semi-supervised initialization (SSI) that allows the agent to learn from var- ious possible hints before training under dif- ferent tasks. Experiments show that SSI not only helps to learn faster (1.2x) but also has a higher success rate (11% relative improve- ment) of the final policy. 1 Introduction Most Reinforcement Learning (RL) methods (Mnih et al., 2013, 2016) rely on an agent to explore and maximize the feedback reward. Since designing a reward for each step is impractical, a common setting is only to give out the achieved signal. In Atari Grand Challenge (Kurin et al., 2017), only if achieving the goal, the environmental reward is 1. However, this sparse-reward setting makes the agent difficult to learn (Vecerik et al., 2017). ExtLang (Goyal et al., 2019) incorporates lan- guage hints as an additional reward to overcome the sparse-reward issue. They first build 45 different tasks under Montezuma’s Revenge game, where each task consists of a starting state, an unknown goal position, and a given hint. They also col- lect demo clips, which are partial playing records (states and actions) that each corresponds to a hint, as shown in Fig. 1. To provide an additional reward, ExtLang pre-trains a reward module to reflect the relevance between agent actions and the given hint when exploring a task. In this way, they can supply a hint reward instead of only the sparse environ- mental reward to make the agent easier to explore. Though providing an additional reward, the hints are usually incomplete (Kuhlmann et al., 2004). Considering a task in Fig. 1, to achieve the goal, Figure 1: For a task with hint “jump over the skull", both “climb up the ladder" and “jump to get the key" are useful latent hints and can be learned during SSI. the agent should “jump over the skull," “climb up the ladder," and “jump to get the key." However, the given hint only contains the first one, and learn- ing those latent hints still relies on the sparse en- vironmental reward. To deal with this issue, we propose semi-supervised initialization (SSI) that enables the agent to experience various possible hints in advance. We adopt a hint module to gener- ate possible hints for random states and allow the agent to learn from them. With SSI, agents have a better-initialized policy during training for each task. From another point of view, in …

Content truncated to 500 words