Human-centric dialog training via offline reinforcement learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3985–4003, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3985 Human-centric dialog training via offline reinforcement learning Natasha Jaques*12, Judy Hanwen Shen*1, Asma Ghandeharioun1, Craig Ferguson1, Agata Lapedriza1, Noah Jones1, Shixiang Shane Gu2, Rosalind Picard1 *Equal contribution 1Massachusetts Institute of Technology, Cambridge, USA <judyshen, asma gh, agata, ncjones, roz>@mit.edu 2Google Research, Mountain View, USA <natashajaques, shanegu>@google.com Abstract How can we train a dialog model to pro- duce better conversations by learning from hu- man feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conver- sations, which we then use to train and im- prove the models using offline reinforcement learning (RL). We identify implicit conversa- tional cues including language similarity, elici- tation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well- known challenge is that learning an RL pol- icy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future re- ward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre- trained prior language model, and use a new strategy to make the algorithm pessimistic, in- stead of optimistic, in the face of uncertainty. We test the resulting dialog model with rat- ings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improv- ing any existing generative dialog model using a static dataset of human feedback. 1 Introduction Training open-domain dialog models is inherently difficult, since for each utterance there are many ac- ceptable responses, yet no perfect response. While supervised learning from conversational corpora allows models to learn grammatical structure and even topic coherence, these models do not gener- alize, since the training objectives mostly lead the models to memorize responses within the corpus. Humans are the ultimate authority in evaluating what makes one conversational reply better than another. To learn from real conversations with hu- mans, we created an interactive, online platform which hosted a diverse set of neural network dia- log models that users could chat with in real time. However, when learning from human interactions in the wild it is crucial to be able to learn offline and test the policy before deploying it, lest it learn inap- propriate behaviors (e.g. Horton (2016)). Thus, we need to train and test models offline, to ensure safe model outputs. In order to safely learn to optimize human feedback we pursued an offline reinforce- ment learning approach to training dialog models (see Figure 1). Offline RL is challenging; most deep RL algo- …

Content truncated to 500 words