A Large-Scale Dataset for Empathetic Response Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251–1264 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1251 A Large-Scale Dataset for Empathetic Response Generation Anuradha Welivita, Yubo Xie and Pearl Pu School of Computer and Communication Sciences École Polytechnique Fédérale de Lausanne Switzerland {kalpani.welivita,yubo.xie,pearl.pu}@epfl.ch Abstract Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset. This is especially the case for response generation where emo- tion plays an important role. However, exist- ing empathetic datasets remain small, delaying research efforts in this area, for example, the development of emotion-aware chatbots. One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels. In this paper, we describe a large-scale silver dataset consisting of 1M di- alogues annotated with 32 fine-grained emo- tions, eight empathetic response intents, and the Neutral category. To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually anno- tated data and eventually scaling it to a satis- factory size. We compare its quality against a state-of-the-art gold dataset using offline ex- periments and visual validation methods. The resultant procedure can be used to create sim- ilar datasets in the same domain as well as in other domains.1 1 Introduction Researchers are increasingly inclined towards re- fining pre-trained language models with domain- specific datasets to achieve certain tasks (Devlin et al., 2019; Liu et al., 2019; Rashkin et al., 2018). One such area is the development of empathetic conversational agents that can understand human emotions and respond appropriately. The aim of the empathetic response generation task is to gen- erate syntactically correct, contextually relevant, and more importantly emotionally appropriate re- sponses following previous dialogue turns. Such tasks require the creation and availability of large dialogue datasets, in which each utterance is anno- tated with the correct intents and emotions. Though 1The datasets and the code are publicly accessible at https://github.com/anuradha1992/EDOS. many such datasets have been developed in the past (Busso et al., 2008; Poria et al., 2019; Li et al., 2017; Rashkin et al., 2018), due to the cost of man- ual labor, they are limited in size, thus insufficient to train robust conversational agents. Since collect- ing and manually annotating such gold standard data is expensive, replacing them with automati- cally annotated silver standard data has become a rising interest (Filannino and Di Bari, 2015). We show how such a large-scale silver standard dataset with sufficient quality can be curated and used to fine-tune pre-trained language models for the gen- eration of empathetic responses. Emotions revealed in social chitchat are rather complex. It has many categories of emotions to distinguish due to subtle variations present in hu- man emotion. For example, Sadness and Disap- pointment are pursued and dealt with differently in human conversations even though both of them are negative emotions. Also, the listener’s reaction to emotion is not always a straightforward mirror- ing effect of the speaker’s emotion. Rather …

Content truncated to 500 words