Inferring Rewards from Language in Context

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8546 - 8560 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Inferring Rewards from Language in Context Jessy Lin♦ Daniel Fried♣ Dan Klein♦ Anca Dragan♦ ♦ University of California, Berkeley ♣ Carnegie Mellon University {jessy_lin, klein, anca}@berkeley.edu, dfried@andrew.cmu.edu Abstract In classic instruction following, language like “I’d like the JetBlue flight” maps to actions (e.g., selecting that flight). However, language also conveys information about a user’s under- lying reward function (e.g., a general prefer- ence for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight–booking task with natural language, our model more accurately infers rewards and pre- dicts optimal actions in unseen environments, in comparison to past work that first maps lan- guage to actions (instruction following) and then maps actions to rewards (inverse rein- forcement learning). 1 Introduction Language is a natural interface for systems like robots or personal assistants that interact with hu- man users. One way to interpret language in these interactive settings is to train an instruction fol- lowing agent: a model that learns to map com- mands like “go three steps forward to the door” to a sequence of actions in context (e.g., Brana- van et al. 2009; Tellex et al. 2011, inter alia). In- structions describe how an agent should act in an immediate context, but to build models that can generalize—carrying out a user’s goals in new con- texts and learning user preferences over repeated interactions—agents should also infer why actions are taken. Grounding language to reward func- tions extends the standard instruction following setup in this way, representing the goals and pref- erences that underlie actions, and allowing agents to autonomously carry out correct actions in new contexts (e.g., Fu et al. 2019). However, when people interact with systems they often primarily aim to achieve specific tasks, Available flights: JetBlue Delta price stop length User’s true reward function: User: i’d like the jetblue flight update reward posterior after observing language + context in each interaction price $10 longest stop 2h Delta price $10 longest stop 2h Delta price $60 longest stop 2h Delta price $60 longest stop 2h Delta price $100 longest stop 2h JetBlue price $100 longest stop 2h JetBlue JetBlue Delta price stop length ? ? ? ? ? ? ? ? ... Figure 1: When people instruct agents with lan- guage like “I’d like the JetBlue flight,” both their de- sired actions and the language itself reveal informa- tion about rewards. From the referenced flight itself, a model would guess that the user may prefer expen- sive JetBlue flights. Reasoning jointly with language reveals that JetBlue is the more salient preference, and the model should still have uncertainty about whether expensive flights are …

Content truncated to 500 words