Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5370 Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset Hannah Rashkin1⋆, Eric Michael Smith2, Margaret Li2, Y-Lan Boureau2 1 Paul G. Allen School of Computer Science & Engineering, University of Washington 2 Facebook AI Research hrashkin@cs.washington.edu, {ems,margaretli,ylan}@fb.com Abstract One challenge for dialogue agents is recog- nizing feelings in the conversation partner and replying accordingly, a key communicative skill. While it is straightforward for humans to recognize and acknowledge others’ feelings in a conversation, this is a significant chal- lenge for AI systems due to the paucity of suitable publicly-available datasets for train- ing and evaluation. This work proposes a new benchmark for empathetic dialogue genera- tion and EMPATHETICDIALOGUES, a novel dataset of 25k conversations grounded in emo- tional situations. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by hu- man evaluators, compared to models merely trained on large-scale Internet conversation data. We also present empirical compar- isons of dialogue model adaptations for em- pathetic responding, leveraging existing mod- els or datasets without requiring lengthy re- training of the full model. 1 Introduction A desirable trait in a human-facing dialogue agent is to appropriately respond to a conversa- tion partner that is describing personal experi- ences, by understanding and acknowledging any implied feelings — a skill we refer to as em- pathetic responding. For instance, while the crossed-out response in Figure 1 is topically rel- evant, “Congrats! That’s great!” may be more satisfying because it acknowledges the underly- ing feelings of accomplishment in an empathetic way. In this work, we investigate empathetic response generation from current dialogue sys- tems, and propose experiments using a new re- source, EMPATHETICDIALOGUES, as a bench- mark to evaluate this skill set. ⋆This work was done while first author was intern at Facebook AI Research (FAIR). I finally got promoted today at work. Why would anyone promote you? Congrats! That’s great! feels proud 😀 EMPATHETICDIALOGUES dataset example Listener Speaker Figure 1: Example where acknowledging an inferred feeling is appropriate Empathetic responding is clearly relevant to dialogue systems that are geared towards gen- eral conversation or chit-chat. Indeed, ordinary communication is frequently prompted by peo- ple sharing their feelings or circumstances. But researchers analyzing goal-directed conversations have also observed the frequent intrusion of or- dinary conversation in those interactions as well, either as a “warm-up” introduction or as a de- tour (Levinson et al., 2000; Heritage, 2005). En- gaging in social talk, reacting to emotional cues and displaying a caring attitude have, in fact, been associated with better task outcomes in many domains (Wentzel, 1997; Levinson et al., 2000; Bickmore and Cassell, 2001; Kim et al., 2004; Fraser et al., 2018). While many of those stud- ies deal with human-human interactions, humans have been shown to often interact with machines in a natural and social way (Reeves and Nass, 1996; …

Content truncated to 500 words