Environment-Driven Lexicon Induction for High-Level Instructions

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 992–1002, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Environment-Driven Lexicon Induction for High-Level Instructions Dipendra K. Misra∗ Kejia Tao∗ Percy Liang∗∗ Ashutosh Saxena∗ dkm@cs.cornell.edu kt454@cornell.edu pliang@cs.stanford.edu asaxena@cs.cornell.edu ∗ Department of Computer Science, Cornell University ∗∗Department of Computer Science, Stanford University Abstract We focus on the task of interpreting com- plex natural language instructions to a robot, in which we must ground high-level commands such as microwave the cup to low-level actions such as grasping. Pre- vious approaches that learn a lexicon dur- ing training have inadequate coverage at test time, and pure search strategies can- not handle the exponential search space. We propose a new hybrid approach that leverages the environment to induce new lexical entries at test time, even for new verbs. Our semantic parsing model jointly reasons about the text, logical forms, and environment over multi-stage instruction sequences. We introduce a new dataset and show that our approach is able to suc- cessfully ground new verbs such as dis- tribute, mix, arrange to complex logical forms, each containing up to four predi- cates. 1 Introduction The task of mapping natural language instructions to actions for a robot has been gaining momen- tum in recent years (Artzi and Zettlemoyer, 2013; Tellex et al., 2011; Misra et al., 2014; Bollini et al., 2011; Guadarrama et al., 2013; Matuszek et al., 2012b; Fasola and Mataric, 2013). We are particularly interested in instructions contain- ing verbs such as “microwave” denoting high-level concepts, which correspond to more than 10 low- level symbolic actions such as grasp. In this setting, it is common to find new verbs requiring new concepts at test time. For example, in Fig- ure 1, suppose that we have never seen the verb “fill”. Can we impute the correct interpretation, and moreover seize the opportunity to learn what “fill” means in a way that generalizes to future in- structions? Text: “get the cup, fill it with water and then microwave the cup” grasping cup3 ∧ near(robot1,cup3) in cup3,microwave ∧ state(microwave1,is-on) state cup3,water ∧ on(cup3,sink) Unseen verb “ fill ” is grounded at test time using environment. Figure 1: A lexicon learned on the training data cannot possibly cover all the verb-concept map- pings needed at test time. Our algorithm learns the meaning of new verbs (e.g., “fill”) using the environment context. Previous work in semantic parsing handles lex- ical coverage in one of two ways. Kwiatkowski et al. (2010) induces a highly constrained CCG lex- icon capable of mapping words to complex log- ical forms, but it would have to skip new words (which in Figure 1 would lead to microwaving an empty cup). Others (Berant and Liang, 2014) take a freer approach by performing a search over log- ical forms, which can handle new words, but the logical forms there are much simpler than the ones we consider. In this paper, we present …

Content truncated to 500 words