Improving Natural Language Interaction with Robots Using Advice

Proceedings of NAACL-HLT 2019, pages 1962–1967 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1962 Improving Natural Language Interaction with Robots Using Advice Nikhil Mehta Department of Computer Science Purdue University West Lafayette, IN 47907 mehta52@purdue.edu Dan Goldwasser Department of Computer Science Purdue University West Lafayette, IN 47907 dgoldwas@purdue.edu Abstract Over the last few years, there has been grow- ing interest in learning models for physically grounded language understanding tasks, such as the popular blocks world domain. These works typically view this problem as a single- step process, in which a human operator gives an instruction and an automated agent is eval- uated on its ability to execute it. In this pa- per we take the first step towards increasing the bandwidth of this interaction, and suggest a protocol for including advice, high-level ob- servations about the task, which can help con- strain the agents prediction. We evaluate our approach on the blocks world task, and show that even simple advice can help lead to sig- nificant performance improvements. To help reduce the effort involved in supplying the ad- vice, we also explore model self-generated ad- vice which can still improve results. 1 Introduction The problem of constructing an artificial agent ca- pable of understanding and executing human in- structions is one of the oldest long-standing AI challenges (Winograd, 1972). This problem has numerous applications in various domains (plan- ning, navigation and assembly) and can help ac- commodate seamless interaction with personal as- sistants in many environments. Due to its central role in AI and wide applicability, this problem has seen a surge of interest recently (MacMahon et al., 2006; Branavan et al., 2009; Chen and Mooney, 2011; Tellex et al., 2011; Matuszek et al., 2012; Kim and Mooney, 2013; Misra et al., 2017). Recent works (Bisk et al., 2016; Tan and Bansal, 2018) focus on exploring deep learning methods for grounding spatial language. In this popular setup, human communication with robots is viewed as a single-step process, in which a nat- ural language (NL) instruction is provided, and an outcome is observed. The target is in the lower left. Locate the top-most block and place it directly below the right- most tower. x Figure 1: Based on the instruction (upper sentence) the model predicts the coordinates of the block and its tar- get location. The ‘x’ represents an incorrect prediction, corrected by the provided advice (lower sentence). Our goal in this paper is to explore different ap- proaches for relaxing the single step assumption, and present initial results which we hope would motivate future work in this direction. Similar to interactive dialog systems (Allen et al., 1995; Ryb- ski et al., 2007; Wen et al.), we view this problem as an interactive process, in which the human op- erator can observe the agents’ response to their in- struction and adjust it by providing advice, a form of online feedback. Specifically, the advice con- sists of a short sentence, simplifying the user’s in- tent. We …

Content truncated to 500 words