DANLI: Deliberative Agent for Following Natural Language Instructions

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1280 - 1298 December 7-11, 2022 ©2022 Association for Computational Linguistics DANLI: Deliberative Agent for Following Natural Language Instructions Yichi Zhang Jianing Yang Jiayi Pan Shane Storks Nikhil Devraj Ziqiao Ma Keunwoo Peter Yu Yuwei Bao Joyce Chai Computer Science and Engineering Division, University of Michigan zhangyic@umich.edu Abstract Recent years have seen an increasing amount of work on embodied AI agents that can per- form tasks by following human language in- structions. However, most of these agents are reactive, meaning that they simply learn and im- itate behaviors encountered in the training data. These reactive agents are insufficient for long- horizon complex tasks. To address this limita- tion, we propose a neuro-symbolic deliberative agent that, while following language instruc- tions, proactively applies reasoning and plan- ning based on its neural and symbolic represen- tations acquired from past experience (e.g., nat- ural language and egocentric vision). We show that our deliberative agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark. Moreover, the underlying reasoning and planning pro- cesses, together with our modular framework, offer impressive transparency and explainabil- ity to the behaviors of the agent. This enables an in-depth understanding of the agent’s ca- pabilities, which shed light on challenges and opportunities for future embodied agents for instruction following. The code is available at https://github.com/sled-group/DANLI. 1 Introduction Natural language instruction following with embod- ied AI agents (Chai et al., 2018; Anderson et al., 2018; Thomason et al., 2019; Qi et al., 2020; Shrid- har et al., 2020; Padmakumar et al., 2021) is a notoriously difficult problem, where an agent must interpret human language commands to perform actions in the physical world and achieve a goal. Es- pecially challenging is the hierarchical nature of ev- eryday tasks,1 which often require reasoning about 1For example, making breakfast may require preparing one or more dishes (e.g., toast and coffee), each of which requires several sub-tasks of navigating through the environment and manipulating objects (e.g., finding a knife, slicing bread, cook- ing it in the toaster), and even more fine-grained primitive actions entailed by them (e.g., walk forward, pick up knife). subgoals and reconciling them with the world state and overall goal. However, despite recent progress, past approaches are typically reactive (Wooldridge, 1995) in their execution of actions: conditioned on the rich, multimodal inputs from the environ- ment, they perform actions directly without using an explicit representation of the world to facili- tate grounded reasoning and planning (Pashevich et al., 2021; Zhang and Chai, 2021; Sharma et al., 2022). Such an approach is inefficient, as natural language instructions often omit trivial steps that a human may be assumed to already know (Zhou et al., 2021). Besides, the lack of any explicit sym- bolic component makes such approaches hard to interpret, especially when the agent makes errors. Inspired by previous work toward deliberative agents in robotic task planning, which apply long- term action planning over known world and …

Content truncated to 500 words