Empathy Identification Systems are not Accurately Accounting for Context

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1686–1695 May 2-6, 2023 ©2023 Association for Computational Linguistics Empathy Identification Systems are not Accurately Accounting for Context Andrew Lee† Jonathan Kummerfeld†‡ Larry An† Rada Mihalcea† † University of Michigan ‡ University of Sydney ajyl@umich.edu Abstract Understanding empathy in text dialogue data is a difficult, yet critical, skill for effective human-machine interaction. In this work, we ask whether systems are making meaningful progress on this challenge. We consider a sim- ple model that checks if an input utterance is similar to a small set of empathetic examples. Crucially, the model does not look at what the utterance is a response to, i.e., the dialogue con- text. This model performs comparably to prior work on standard benchmarks and even outper- forms state-of-the-art models for empathetic rationale extraction by 16.7 points on T-F1 and 4.3 on IOU-F1. This indicates that current sys- tems rely on the surface form of the response, rather than whether it is suitable in context. To confirm this, we create examples with dialogue contexts that change the interpretation of the response and show that current systems con- tinue to label utterances as empathetic. We dis- cuss the implications of our findings, including improvements for empathetic benchmarks and how our model can be an informative baseline. 1 Introduction Empathy is a fundamental phenomenon that allows us to better communicate and relate with others. Studies show that empathy is significantly corre- lated with counseling treatment outcomes (Moyers and Miller, 2013; Elliott et al., 2018). Computer systems could be improved by adding the ability to understand empathy. EPITOME (Sharma et al., 2020b) took a step towards understanding empathy in language, intro- ducing tasks such as empathy identification and empathetic rationale extraction. Models built us- ing EPITOME have been used to build or evaluate empathetic dialogue systems (Sharma et al., 2021; Zheng et al., 2021; Majumder et al., 2022; Kim et al., 2021) or study social effects of empathy (Chen and Xu, 2021). In this work, we explore whether current models are effectively considering dialogue context (short- ened to context for the rest of this paper). We show that a simple model that does not consider context can achieve strong results, and that a model from prior work does not change its predictions when we make substantial changes to the context. Together, these results indicate that models are more limited than previously thought. We introduce an adapted version of micromod- els (Lee et al., 2021), a simple and explainable approach that combines a set of models, with each model identifying a specific linguistic phenomenon. This approach performs much better than the EPIT- OME’s model on five metrics, comparably on five metrics, and much worse on two. Critically, we achieve this without any use of context. We inspect our model’s behavior and find that it can achieve accuracy scores that are as good as or better than the EPITOME model’s for em- pathetic rationale extraction with as …

Content truncated to 500 words