CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4471 - 4491 July 10-15, 2022 ©2022 Association for Computational Linguistics CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning Siddharth Verma Justin Fu Mengjiao Yang Sergey Levine UC Berkeley vsiddharth@berkeley.edu {justinjfu,sherryy,svlevine}@eecs.berkeley.edu Abstract Conventionally, generation of natural language for dialogue agents may be viewed as a statis- tical learning problem: determine the patterns in human-provided data and generate appro- priate responses with similar statistical proper- ties. However, dialogue can also be regarded as a goal directed process, where speakers at- tempt to accomplish a specific task. Reinforce- ment learning (RL) algorithms are designed specifically for solving such goal-directed prob- lems, but the most direct way to apply RL – through trial-and-error learning in human con- versations, – is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with lan- guage models to yield realistic dialogue agents that better accomplish task goals. 1 Introduction Constructing fluent and intelligent dialogue agents could pave the way for intuitive interfaces and au- tomation of human-interactive tasks. However, this requires dialogue agents that both generate flu- ent, natural responses and effectively pursue the goals of the given dialogue task. A predominant approach to training dialogue agents is through su- pervised learning, where an agent is tasked with imitating language provided by humans. While this can provide for fluent responses, it becomes difficult to ensure that such agents systematically pursue the goals of the dialogue interaction. If we instead view dialogue as a control problem, frame- works such as reinforcement learning (RL) could allow agents to automatically optimize dialogue with respect to a task goal through a trial-and-error process and improve over human behavior. However, implementing an RL system in prac- tice, where an agent learns online from interact- ing with real humans, can be prohibitively expen- sive and time-consuming. This is in stark contrast to supervised learning approaches, where we can cheaply construct datasets for training imitation agents by simply logging conversations. There- fore, existing RL approaches for dialogue often rely on interacting with a learned model of a hu- man (Li et al., 2016b; He et al., 2018), from which experience can be generated inexpensively. How- ever, naïve training in this manner can result in the dialogue agent exploiting the model, which can de- generate into non-intelligible language. To mitigate this, algorithms must typically enforce strong pri- ors to keep generated language similar to those seen in the dataset (Li et al., 2016b; Jaques et al., 2019), or adopt dialogue management and template-based approaches which directly re-use language seen in the dataset (He et al., 2018). Issues such as model exploitation and distribu- tion shift when training on static datasets are a primary concern of offline RL (Levine et al., 2020), and can …

Content truncated to 500 words