SRL4E – Semantic Role Labeling for Emotions: A Unified Evaluation Framework

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4586 - 4601 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SRL4E – Semantic Role Labeling for Emotions: A Unified Evaluation Framework Cesare Campagnano1 and Simone Conia1 and Roberto Navigli2 Sapienza NLP Group 1Department of Computer Science 2Department of Computer, Control and Management Engineering Sapienza University of Rome {campagnano,conia}@di.uniroma1.it navigli@diag.uniroma1.it Abstract In the field of sentiment analysis, several stud- ies have highlighted that a single sentence may express multiple, sometimes contrasting, senti- ments and emotions, each with its own expe- riencer, target and/or cause. To this end, over the past few years researchers have started to collect and annotate data manually, in order to investigate the capabilities of automatic sys- tems not only to distinguish between emotions, but also to capture their semantic constituents. However, currently available gold datasets are heterogeneous in size, domain, format, splits, emotion categories and role labels, making comparisons across different works difficult and hampering progress in the area. In this paper, we tackle this issue and present a uni- fied evaluation framework focused on Seman- tic Role Labeling for Emotions (SRL4E), in which we unify several datasets tagged with emotions and semantic roles by using a com- mon labeling scheme. We use SRL4E as a benchmark to evaluate how modern pretrained language models perform and analyze where we currently stand in this task, hoping to pro- vide the tools to facilitate studies in this com- plex area. 1 Introduction Emotion detection – a long-standing open prob- lem in Natural Language Processing (NLP) – is the task of automatically associating one or more emo- tions with a text. Even though emotional states are highly subjective and often depend on several fac- tors, such as one’s past experiences, culture and ed- ucation, the automatic identification, categorization and analysis of emotions in texts has been found to be beneficial in a wide array of downstream tasks, such as hate speech detection (Markov et al., 2021), sarcasm detection (Chauhan et al., 2020), and modeling political discourse (Huguet Cabot et al., 2021), inter alia. In the past decade, Deep Learning techniques have become ubiquitous in the development of au- tomatic systems for an increasing number of NLP tasks, including emotion detection (Chatterjee et al., 2019). However, most of the effective neural-based approaches still require significant amounts of train- ing data in order to learn to perform at their best. For this reason, with a view to bootstrapping the development of neural systems for emotion detec- tion, there have been several efforts to annotate cor- pora with emotions manually (Bostan and Klinger, 2018). Nevertheless, over the past few years, numer- ous studies have indicated that a short text, even a single sentence, may contain multiple – at times concurring, at other times contrasting – sentiments and emotions. And not only this, two emotions in the same sentence may be experienced, tar- geted, and/or caused by different semantic con- stituents which, similarly to predicate-argument structures …

Content truncated to 500 words