AVAST: Attentive Variational State Tracker in a Reinforced Navigator

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 424–433 November 20–23, 2022. ©2022 Association for Computational Linguistics 424 AVAST: Attentive Variational State Tracker in a Reinforced Navigator Je-Wei Jang1, Mahdin Rohmatillah2, and Jen-Tzung Chien1 1Institute of Electrical and Computer Engineering 2EECS International Graduate Program National Yang Ming Chiao Tung University, Taiwan {carbon1124.ee08,mahdin.ee08,jtchien}@nycu.edu.tw Abstract Recently, emerging approaches have been pro- posed to deal with robotic navigation problems, especially vision-and-language navigation task which is one of the most realistic indoor navi- gation challenge tasks. This task can be mod- elled as a sequential decision-making problem, which is suitable to be solved by deep reinforce- ment learning. Unfortunately, the observations provided from the simulator in this task are not fully observable states, which exacerbate the difficulty of implementing reinforcement learning. To deal with this challenge, this pa- per presents a novel method, called as attentive variational state tracker (AVAST), a variational approach to approximate belief state distribu- tion for the construction of a reinforced navi- gator. The variational approach is introduced to improve generalization to the unseen envi- ronment which barely achieved by traditional deterministic state tracker. In order to stabilize the learning procedure, a fine-tuning process using policy optimization is proposed. From the experimental results, the proposed AVAST does improve the generalization relative to pre- vious works in vision-and-language navigation task. A significant performance is achieved without requiring any additional exploration in the unseen environment.1 1 Introduction Reinforcement learning (RL) has become a cru- cial and successful solution in many sequential decision-making problems, such as video game playing AI (Bellemare et al., 2013) and robotic control (Todorov et al., 2012). In theory, RL al- gorithms are designed for solving problems under the assumption of Markov decision process (MDP), which means that the observation provided from the environment needs to exactly represent the com- plete state information of the environment (Chien 1The dataset, simulator and training code are publicly avail- able at: https://github.com/NYCU-MLLab/ et al., 2021). However, most of the real-world problems, such as bridge-playing AI, dialogue sys- tems (Rohmatillah and Chien, 2021b; Hsu et al., 2021; Rohmatillah and Chien, 2021a), autonomous driving, and first-person navigation (Kempka et al., 2016), can not be directly modeled as Markov de- cision processes, because of the incomplete state information. For example, in dialogue task, sys- tem does not have an access to the user goal (Jang et al., 2022). In order to improve the generaliza- tion, partially observable Markov decision process (POMDP) (Åström, 1965) was designed to model the process in which the agent does not have access to observe complete state information. In case of vision-and-language navigation (VLN) task, the problem formulation is considered as POMDP problem, as the agent does not receive full information about the state. It only receive the information about the images of surroundings and the texts which describe the navigation task and agent pose information. There is no …

Content truncated to 500 words