Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2445–2453 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2445 Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback Makesh Narsimhan Sreedhar Mila Universite de Montreal narsimma@mila.quebec Kun Ni Mila Universite de Montreal nikun@mila.quebec Siva Reddy Facebook CIFAR AI Chair, Mila McGill University siva@cs.mcgill.ca Abstract The ubiquitous nature of chatbots and their interaction with users generate an enormous amount of data. Can we improve chatbots using this data? A self-feeding chatbot im- proves itself by asking natural language feed- back when a user is dissatisfied with its re- sponse and uses this feedback as an additional training sample. However, user feedback in most cases contains extraneous sequences hin- dering their usefulness as a training sample. In this work, we propose a generative adversar- ial model that converts noisy feedback into a plausible natural response in a conversation. The generator’s goal is to convert the feed- back into a response that answers the user’s previous utterance and to fool the discrimina- tor which distinguishes feedback from natural responses. We show that augmenting original training data with these modified feedback re- sponses improves the original chatbot perfor- mance from 69.94% to 75.96% in ranking cor- rect responses on the PERSONACHAT dataset, a large improvement given that the original model is already trained on 131k samples.1 1 Introduction Enabling chatbots to indulge in engaging conver- sations requires massive datasets of human-human conversations (Ritter et al., 2011; Sordoni et al., 2015; Vinyals and Le, 2015; Zhang et al., 2018, 2019). Training such dialog agents requires sub- stantial time and effort expended in the collection of adequate number of high quality conversation samples. Hancock et al. (2019) alleviate this prob- lem by introducing a self-feeding chatbot which can directly learn from user interactions. This chat- bot requests users to provide natural language feed- back when the users are dissatisfied with its re- 1Our code is released at https://github.com/ ekunnii/adversarial-feedback-chatbot/ You could have talked about your age. Feed2Resp Hey, I am 20. How old are you? Response What is your favourite movie? My favourite movie is Toy Story. How old are you? I like hiking. Feedback Conversation History Oops, I messed up. What should I have said? New Training Sample Retrain chatbot Figure 1: When the bot provides a poor response to the question posed by the user, the bot requests natural lan- guage feedback. We use the conversation context and the feedback to construct a plausible response to the user query and use it as an additional training sample to improve the chatbot. sponse. Hancock et al. treat this feedback as a gold response to the wrong turn and use it as an additional training sample to improve the chatbot. Although natural language feedback is cheap to collect from a chatbot’s end-users, most often, feed- back cannot be used directly as a training sample since feedback is usually not the answer itself, but simply contains hints to the answer. Table 1 shows …

Content truncated to 500 words