Skill Induction and Planning with Latent Language

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1713 - 1726 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Skill Induction and Planning with Latent Language Pratyusha Sharma Antonio Torralba Jacob Andreas Massachusetts Institute of Technology {pratyuss,torralba,jda}@mit.edu Abstract We present a framework for learning hierarchi- cal policies from demonstrations, using sparse natural language annotations to guide the discovery of reusable skills for autonomous decision-making. We formulate a generative model of action sequences in which goals generate sequences of high-level subtask de- scriptions, and these descriptions generate se- quences of low-level actions. We describe how to train this model using primarily unan- notated demonstrations by parsing demonstra- tions into sequences of named high-level sub- tasks, using only a small number of seed anno- tations to ground language in action. In trained models, natural language commands index a combinatorial library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals. We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations. It achieves task completion rates comparable to state-of-the-art models (outperforming several recent methods with access to ground-truth plans during training and evaluation) while providing structured and human-readable high-level plans.1 1 Introduction Building autonomous agents that integrate high- level reasoning with low-level perception and con- trol is a long-standing challenge in artificial intelli- gence (Fikes et al., 1972; Newell, 1973; Sacerdoti, 1973; Brockett, 1993). Fig. 1 shows an example: to accomplish a task such as cooking an egg, an agent must first find the egg, then grasp it, then locate a stove or microwave, at each step reasoning about both these subtasks and complex, unstructured sen- sor data. Hierarchical planning models (e.g. Sut- ton et al., 1999)—which first reason about abstract 1Code and visualizations: https://sites.google.com/ view/skill-induction-latent-lang/. Annotated demonstrations (10%) Unannotated demonstrations (90%) Training: Semi-supervised Skill Learning with Latent Language Deployment : Planning with Language Pick up an egg. … Heat and cool an egg. grasp(ob1) open(ob3) turn(left) Grab the ladle. Go to the sink. … Put a clean ladle on the counter turn(right) forward grasp(obj1) Put the ladle down. … … Find a knife. Place tomato in the fridge. Slice and chill a tomato. forward grasp(ob4) turn(right) … Goal Plan Heat the egg in the microwave. Alignments Actions Model Architecture Find an egg. embed cond. LM open(ob3) observation action mask Slice the tomato. Find a tomato. Language Model Planner cond. LM Policy observed Legend inferred / predicted Figure 1: Hierarchical imitation learning using weak natural language supervision. During training, a small number of seed annotations are used to automatically segment and label unannotated training demonstrations with natural language descriptions of their high-level structure. When deployed on new tasks, learned poli- cies first generate sequences of natural language sub- task descriptions, then modularly translate each de- scription to a sequence of low-level actions. states and actions, then ground these in concrete control decisions—play a key role in most …

Content truncated to 500 words