Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Transactions of the Association for Computational Linguistics, 1 (2013) 49–62. Action Editor: Jason Eisner. Submitted 11/2012; Published 3/2013. c⃝2013 Association for Computational Linguistics. Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions Yoav Artzi and Luke Zettlemoyer Computer Science & Engineering University of Washington Seattle, WA 98195 {yoav,lsz}@cs.washington.edu Abstract The context in which language is used pro- vides a strong signal for learning to recover its meaning. In this paper, we show it can be used within a grounded CCG semantic parsing approach that learns a joint model of mean- ing and context for interpreting and executing natural language instructions, using various types of weak supervision. The joint nature provides crucial benefits by allowing situated cues, such as the set of visible objects, to di- rectly influence learning. It also enables algo- rithms that learn while executing instructions, for example by trying to replicate human ac- tions. Experiments on a benchmark naviga- tional dataset demonstrate strong performance under differing forms of supervision, includ- ing correctly executing 60% more instruction sets relative to the previous state of the art. 1 Introduction The context in which natural language is used pro- vides a strong signal to reason about its meaning. However, using such a signal to automatically learn to understand unrestricted natural language remains a challenging, unsolved problem. For example, consider the instructions in Figure 1. Correct interpretation requires us to solve many sub- problems, such as resolving all referring expres- sions to specific objects in the environment (includ- ing, “the corner” or “the third intersection”), disam- biguating word sense based on context (e.g., “the chair” could refer to a chair or sofa), and finding executable action sequences that satisfy stated con- straints (such as “twice” or “to face the blue hall”). move forward twice to the chair λa.move(a) ∧ dir(a, forward) ∧ len(a, 2) ∧ to(a, ιx.chair(x)) at the corner turn left to face the blue hall λa.pre(a, ιx.corner(x)) ∧ turn(a) ∧ dir(a, left) ∧ post(a, front(you, ιx.blue(x) ∧ hall(x))) move to the chair in the third intersection λa.move(a) ∧ to(a, ιx.sofa(x)) ∧ intersect(order(λy.junction(y), frontdist, 3), x) Figure 1: A sample navigation instruction set, paired with lambda-calculus meaning representations. We must also understand implicit requests, for ex- ample from the phrase “at the corner,” that describe goals to be achieved without specifying the specific steps. Finally, to do all of this robustly without pro- hibitive engineering effort, we need grounded learn- ing approaches that jointly reason about meaning and context to learn directly from their interplay, with as little human intervention as possible. Although many of these challenges have been studied separately, as we will review in Section 3, this paper represents, to the best of our knowledge, the first attempt at a comprehensive model that ad- dresses them all. Our approach induces a weighted Combinatory Categorial Grammar (CCG), includ- ing both the parameters of the linear model and a CCG lexicon. To model complex instructional lan- guage, we introduce a new semantic modeling ap- proach that can represent …

Content truncated to 500 words