Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4202–4213 August 1–6, 2021. ©2021 Association for Computational Linguistics 4202 Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring Yichi Zhang Joyce Y. Chai Computer Science and Engineering University of Michigan Ann Arbor, MI, USA {zhangyic, chaijy}@umich.edu Abstract Despite recent progress, learning new tasks through language instructions remains an ex- tremely challenging problem. On the AL- FRED benchmark for task learning, the pub- lished state-of-the-art system only achieves a task success rate of less than 10% in an un- seen environment, compared to the human per- formance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-to- end architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT1 (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a uni- fied manner to learn a hierarchical task struc- ture. On the ALFRED benchmark, HiTUT has achieved the best performance with a remark- ably higher generalization ability. In the un- seen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit rep- resentation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark develop- ment and evaluation. 1 Introduction As physical agents (e.g., robots) start to emerge as our assistants and partners, it has become increas- ingly important to empower these agents with an ability to learn new tasks by following human lan- guage instructions. Many benchmarks have been developed to study the agent’s ability to follow natural language instructions in various domains including navigation (Anderson et al., 2018; Chen et al., 2019), object manipulation (Misra et al., 1Source code available at https://github.com/ 594zyc/HiTUT Put(Mug, Sink) TurnOn(Faucet) TurnOff(Faucet) Goal Directive Place a clean mug in the coffee machine. Goto(Mug) Pickup(Mug) Goto(Sink) Clean(Mug) Goto( CoffeeMachine) Put (Mug, Coffeemachine) RotateLeft RotateLeft MoveAhead MoveAhead … Pickup(Mug) End Sub-Goal Instruction: Go back towards the table. Sub-Goal Instruction: Pick up the dirty mug. … Sub-Goal Instruction: Wash the mug in the sink … … … Sub-Goal Planning Scene Navigation Object Manipulation Figure 1: An example task in ALFRED. 2017; Zhu et al., 2017) and embodied reasoning (Das et al., 2018a; Gordon et al., 2018). Despite re- cent progress, learning new tasks through language instructions remains an extremely challenging prob- lem as it touches upon almost every aspect of AI from perception, reasoning, to planning and actions. For example, on the ALFRED benchmark for task learning (Shridhar et al., 2020), the state-of-the-art system only achieves less than 10% task success rate in an unseen environment (Singh et al., 2020), compared to the human performance of over 90%. Most previous works apply an end-to-end neural ar- chitecture (Shridhar et al., 2020; Singh et al., 2020; Storks et al., 2021) which attempt to map language …

Content truncated to 500 words