Alignment-Based Compositional Semantics for Instruction Following

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1165–1174, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Alignment-Based Compositional Semantics for Instruction Following Jacob Andreas and Dan Klein Computer Science Division University of California, Berkeley {jda,klein}@cs.berkeley.edu Abstract This paper describes an alignment-based model for interpreting natural language in- structions in context. We approach in- struction following as a search over plans, scoring sequences of actions conditioned on structured observations of text and the environment. By explicitly modeling both the low-level compositional structure of individual actions and the high-level struc- ture of full plans, we are able to learn both grounded representations of sentence meaning and pragmatic constraints on in- terpretation. To demonstrate the model’s flexibility, we apply it to a diverse set of benchmark tasks. On every task, we outperform strong task-specific baselines, and achieve several new state-of-the-art results. 1 Introduction In instruction-following tasks, an agent executes a sequence of actions in a real or simulated envi- ronment, in response to a sequence of natural lan- guage commands. Examples include giving nav- igational directions to robots and providing hints to automated game-playing agents. Plans speci- fied with natural language exhibit compositional- ity both at the level of individual actions and at the overall sequence level. This paper describes a framework for learning to follow instructions by leveraging structure at both levels. Our primary contribution is a new, alignment- based approach to grounded compositional se- mantics. Building on related logical approaches (Reddy et al., 2014; Pourdamghani et al., 2014), we recast instruction following as a pair of nested, structured alignment problems. Given instructions and a candidate plan, the model infers a sequence- to-sequence alignment between sentences and atomic actions. Within each sentence–action pair, the model infers a structure-to-structure alignment between the syntax of the sentence and a graph- based representation of the action. At a high level, our agent is a block-structured, graph-valued conditional random field, with align- ment potentials to relate instructions to actions and transition potentials to encode the environment model (Figure 3). Explicitly modeling sequence- to-sequence alignments between text and actions allows flexible reasoning about action sequences, enabling the agent to determine which actions are specified (perhaps redundantly) by text, and which actions must be performed automatically (in or- der to satisfy pragmatic constraints on interpreta- tion). Treating instruction following as a sequence prediction problem, rather than a series of inde- pendent decisions (Branavan et al., 2009; Artzi and Zettlemoyer, 2013), makes it possible to use general-purpose planning machinery, greatly in- creasing inferential power. The fragment of semantics necessary to com- plete most instruction-following tasks is essen- tially predicate–argument structure, with limited influence from quantification and scoping. Thus the problem of sentence interpretation can reason- ably be modeled as one of finding an alignment be- tween language and the environment it describes. We allow this structure-to-structure alignment— an “overlay” of language onto the world—to be mediated by linguistic structure (in the form of dependency parses) and structured perception (in what we …

Content truncated to 500 words