Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 669–683 October 12–17, 2022. 669 Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation Young-Jun Lee Chae-Gyun Lim School of Computing, KAISTHo-Jin Choi {yj2961,rayote,hojinc}@kaist.ac.kr Abstract Since empathy plays a crucial role in increas- ing social bonding between people, many stud- ies have designed their own dialogue agents to be empathetic using the well-established method of fine-tuning. However, they do not use prompt-based in-context learning, which has shown powerful performance in various natural language processing (NLP) tasks, for empathetic dialogue generation. Although several studies have investigated few-shot in- context learning for empathetic dialogue gen- eration, an in-depth analysis of the genera- tion of empathetic dialogue with in-context learning remains unclear, especially in GPT- 3 (Brown et al., 2020). In this study, we explore whether GPT-3 can generate empathetic dia- logues through prompt-based in-context learn- ing in both zero-shot and few-shot settings. To enhance performance, we propose new in-context example selection methods, called SITSM and EMOSITSM, that utilize emo- tion and situational information. We also in- troduce a new automatic evaluation method, DIFF-EPITOME, which reflects the human ten- dency to express empathy. From the analy- sis, we reveal that our DIFF-EPITOME is ef- fective in measuring the degree of human empathy. We show that GPT-3 achieves competitive performance with Blender 90M, a state-of-the-art dialogue generative model, on both automatic and human evaluation. Our code is available at https://github. com/passing2961/EmpGPT-3. 1 Introduction Empathy refers to the ability to understand another person’s experiences and feelings. This is impor- tant for increasing social bonding (rapport) with conversation partners (Zech and Rimé, 2005). Em- pathy is a multi-dimensional concept consisting of two main aspects: cognitive and affective (Davis et al., 1980). Since Rashkin et al. (2018) released the EMPATHETICDIALOGUES dataset for empa- thetic dialogue generation task, previous studies have improved their dialogue agents to generate more empathetic dialogues (Lin et al., 2019; Ma- jumder et al., 2020; Zheng et al., 2021; Kim et al., 2021b; Sabour et al., 2021; Li et al., 2022). De- pending on how the dialogue agents are trained, these approaches are largely divided into two cat- egories depending on how to train own dialogue agents: (i) training from scratch or (ii) fine-tuning a pretrained dialogue generative model. However, neither of these approaches uses the prompt-based in-context learning paradigm in zero-shot and few- shot settings. Recently, many researchers have attempted to build large-scale language models (LLMs), such as GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022), and HyperCloVA (Kim et al., 2021a). These models have shown surprising performance in vari- ous NLP tasks via prompt-based in-context learn- ing, which is a new paradigm learning technique. Previous studies have explored the effect of few- shot in-context learning on dialogue generation tasks (Zheng and Huang, 2021; Madotto et al., 2021). While Madotto et al. (2021) explored few- shot in-context learning for empathetic dialogue generation, they did not conduct an in-depth analy- sis of …

Content truncated to 500 words