Beyond Goldfish Memory: Long-Term Open-Domain Conversation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5180 - 5197 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Beyond Goldfish Memory∗: Long-Term Open-Domain Conversation Jing Xu Arthur Szlam Jason Weston Facebook AI Research New York, NY {jingxu23,aszlam,jase}@fb.com Abstract Despite recent improvements in open-domain dialogue models, state-of-the-art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human- human dataset consisting of multiple chat ses- sions whereby the speaking partners learn about each other’s interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term con- versation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall pre- vious conversations outperform the standard encoder-decoder architectures currently consid- ered state-of-the-art. 1 Introduction Improvements in the ability to train large neural language models, together with the availability of larger and higher quality dialogue datasets, are spurring the development of increasingly convinc- ing open-domain dialogue models (McTear, 2020). Unfortunately, a major aspect missing from the cur- rent state of the art is that human conversations can take place over long time frames, whereas the currently used systems suffer in this setting. Commonly used training and evaluation resources – while large in terms of number of training exam- ples – include only short conversations, typically between 2-15 turns, consisting of a single conversa- tional session. Perhaps for that reason, the current state-of-the-art models such as Meena (Adiwar- dana et al., 2020) and BlenderBot (Roller et al., 2020) employ Transformers with token truncation lengths of only the 128 most recent tokens, and ∗We use this term colloquially, see Agranoff et al. (1965) for evidence of goldfish long-term memory. are clearly incapable of incorporating long-term conversational context. Consequently, it is unclear how well these models will perform on long or multi-session open-domain conversations. In con- trast, a successfully deployed bot will engage in many conversations over a length of time, as cap- turing organic user interest will garner continual reengagement from returning users. Long-term open-domain communication gives the opportunity for the conversation to develop and even improve with time as the model has more context and more understanding of that specific user’s interests. How- ever current models, due to context truncation, will never use this information. In this work we study methods for long-term open-domain conversation. As to the best of our knowledge no public domain task exists to study such methods, we collect and release1 a new En- glish dataset, entitled Multi-Session Chat (MSC) that consists of human-human crowdworker chats over 5 sessions, with each session consisting of up to 14 utterances, where the conversationalists reengage after a number of hours or days and con- tinue chatting. Previous sessions are annotated …

Content truncated to 500 words