Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13557–13572 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback Jing Xu Meta AI Megan Ung Meta AI Mojtaba Komeili Meta AI Kushal Arora Meta AI & Mila / McGill University Y-Lan Boureau Meta AI Jason Weston Meta AI Abstract Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect de- ployment data, which we make publicly avail- able, of human interactions, and collect vari- ous types of human feedback – including bi- nary quality measurements, free-form text feed- back, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard su- pervised learning, rejection sampling, model- guiding and reward-based learning, in order to make recommendations on which type of feed- back and algorithms work best. We find the recently introduced DIRECTOR model (Arora et al., 2022) shows significant improvements over other existing approaches. 1 Introduction Large language models employed as dialogue agents are primarily trained on human-written documents and human-human conversations col- lected from the web for pre-training (Conneau et al., 2019; Baumgartner et al., 2020), and human- human crowdsourced conversations (Smith et al., 2020) for fine-tuning. The models are then used at inference time to conduct conversations with humans, with no further learning taking place (Adi- wardana et al., 2020; Roller et al., 2020). Human- model conversations – which are never seen at train- ing time – can have a quite different distribution to the original human-human training data used, and our current techniques can lose performance due to lack of robustness to such deviations (Chollet, 2019; Bengio, 2019). Search: F1 metric Search Engine Retrieved Documents Input: What’s happening in F1 these days? Response: F1 is a metric used for classification. Human Feedback Types Good? or Bad? Bad Search? Bad Knowledge? Bad Final Response? Explanation what’s wrong (free-form feedback): Search was the wrong type of F1, not racing! : Better suggestion? Formula 1 news x x Figure 1: Using human feedback to improve open- domain internet-driven dialogue agents. We compare various types of feedback (and corresponding learn- ing algorithms) in this work, such as binary feedback (good/bad), free-form text or supervised responses (bet- ter suggestions) for different modules of the system. In this work, we study learning from the feed- back collected during deployment of models in human-model conversations. Such a setting has the opportunity to learn from within-distribution data, both in terms of the input contexts, but also the responses required (targets). Not only can this mean improvement in skills that are similar to the pre-train and fine-tune data, but potentially the learning of completely …

Content truncated to 500 words