Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior Noriyuki Kojima, Alane Suhr, Yoav Artzi Department of Computer Science and Cornell Tech, Cornell University, USA nk654@cornell.edu {suhr, yoav}@cs.cornell.edu Abstract We study continual learning for natural lan- guage instruction generation, by observing hu- man users’ instruction execution. We focus on a collaborative scenario, where the system both acts and delegates tasks to human users using natural language. We compare user exe- cution of generated instructions to the original system intent as an indication to the system’s success communicating its intent. We show how to use this signal to improve the system’s ability to generate instructions via contex- tual bandit learning. In interaction with real users, our system demonstrates dramatic im- provements in its ability to generate language over time. 1 Introduction Natural language provides an expressive and ac- cessible avenue to instruct non-expert users. The ability to generate instructions is critical for sys- tems that collaborate with users, for example, to delegate tasks. In such scenarios, the system gen- erates language to communicate to the user a latent intent. When users are cooperative and proficient in the language, whether they accomplish the sys- tem’s intent provides an informative, albeit noisy signal of the quality of instruction generation. This implicit signal is fundamentally different from supervised data, including via active learn- ing, in that it does not label the system’s intent with a written instruction, but only provides evidence to the quality of a given instruction in relaying this intent. As a natural byproduct of interaction with users, it also differs from explicit user feedback in not requiring user action beyond what they already do as part of the interaction. Despite its potential and prevalence, this signal is understudied for learning to generate natural language In this paper, we study this learning signal. We formalize continually improving instruction gen- eration by observing human users executing gen- erated instructions. We learn by comparing in- struction execution to the system intent, and demonstrate how this results in a system that con- tinually improves its natural language generation ability through interaction with users. Figure 1 illustrates our learning process. We design a task-oriented collaborative sce- nario using the CEREALBAR game environment (Suhr et al., 2019). In CEREALBAR, two agents, a leader and a follower, work together to complete tasks. The leader plans the tasks to complete, and communicates goals to the follower using natural language. CEREALBAR was originally introduced for studying follower instruction execution. We modify it to focus on generation of leader in- structions, which are then executed by human followers. The collaborative, embodied setup ef- fectively engages users, and aligns their incentives with executing the system’s instructions to the best of their abilities. A major challenge is inferring a learning sig- nal from observed user behavior. Given the user execution, we create positive and negative exam- ples, depending on how the user execution aligns with the system’s plan and the user’s perceived correctness of their own execution. For example, consider an …

Content truncated to 500 words