Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667–3684 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3667 Learning from Dialogue after Deployment: Feed Yourself, Chatbot! Braden Hancock∗ Computer Science Dept. Stanford University bradenjh@cs.stanford.edu Antoine Bordes, Pierre-Emmanuel Mazar´e Jason Weston Facebook AI Research {abordes,pem,jase}@fb.com Abstract The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in con- versation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user’s responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot’s dialogue abilities further. On the PERSONACHAT chit- chat dataset with over 131k training examples, we find that learning from dialogue with a self- feeding chatbot significantly improves perfor- mance, regardless of the amount of traditional supervision. 1 Introduction Training a dialogue agent to converse like a human requires extensive supervision. The most com- mon approach is to train models to imitate hu- mans in large corpora of crowdsourced or scraped conversations (Serban et al., 2015). These fully- supervised conversations tend to be expensive to collect in sufficient quantity and/or occur in set- tings with significant differences from the deploy- ment environment (Ross et al., 2009). Instead, dialogue agents would ideally learn directly from dialogue, the conversations they participate in af- ter deployment, which are usually abundant, task- specific, dynamic, and cheap. This corresponds to the way humans learn to converse—not merely ob- serving others engaging in “expert-level” conver- ∗*BH completed most of this work at Facebook (FAIR). Have you been to France? Yes I have! It’s beautiful. 🤖  Lol. I never eat them! What are you talking about? Oops! I messed up. What should I have said? Maybe ask me what I thought about French food? Satisfaction: 0.85 Satisfaction: 0.10 Extracted Training Examples Context A B C D E F Have you been to France? A Yes, I have! It’s beautiful. Response B Context Have you been to France? A Yes, I have! It’s beautiful. Feedback B Maybe ask me what I thought about French food? F Figure 1: As the self-feeding chatbot engages in dia- logue, it estimates user satisfaction to know when to ask for feedback. From the satisfied responses and feedback responses, new training examples are ex- tracted for the DIALOGUE and FEEDBACK tasks, re- spectively, both of which improve the model’s dialogue abilities further. sations, but instead actively adjusting and correct- ing our speech based on feedback woven through- out our own conversations (Bassiri, 2011; Werts et al., 1995). Giving a dialogue agent this ability would enable it to …

Content truncated to 500 words