EPIC: Multi-Perspective Annotation of a Corpus of Irony

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13844–13857 July 9-14, 2023 ©2023 Association for Computational Linguistics EPIC: Multi-Perspective Annotation of a Corpus of Irony Simona Frenda⋆⊙, Alessandro Pedrani⋄, Valerio Basile⋆, Soda Marem Lo⋆, Alessandra Teresa Cignarella⋆⊙, Raffaella Panizzon⋄, Cristina Marco⋄, Bianca Scarlini⋄, Viviana Patti⋆, Cristina Bosco⋆, Davide Bernardi⋄ ⋆ Computer Science Department, University of Turin, Turin, Italy ⊙ aequa-tech, Turin, Italy ⋄ Alexa AI, Amazon, Amazon Development Centre Italy, Turin, Italy {simona.frenda | valerio.basile | sodamarem.lo | alessandrateresa.cignarella | viviana.patti cristina.bosco}@unito.it {pedrana | panizzor | marcocri | scarlini | dvdbe}@amazon.it Abstract We present EPIC (English Perspectivist Irony Corpus), the first annotated corpus for irony analysis based on the principles of data per- spectivism. The corpus contains short conver- sations from social media in five regional va- rieties of English, and it is annotated by con- tributors from five countries corresponding to those varieties. We analyse the resource along the perspectives induced by the diversity of the annotators, in terms of origin, age, and gender, and the relationship between these dimensions, irony, and the topics of conversation. We vali- date EPIC by creating perspective-aware mod- els that encode the perspectives of annotators grouped according to their demographic char- acteristics. Firstly, the performance of perspec- tivist models confirms that different annotators induce very different models. Secondly, in the classification of ironic and non-ironic texts, per- spectivist models prove to be generally more confident than the non-perspectivist ones. Fur- thermore, comparing the performance on a perspective-based test set with those achieved on a gold standard test set, we can observe how perspectivist models tend to detect more pre- cisely the positive class, showing their ability to capture the different perceptions of irony. Thanks to these models, we are moreover able to show interesting insights about the varia- tion in the perception of irony by the different groups of annotators, such as among different generations and nationalities. 1 Introduction A recent trend in Natural Language Processing (NLP) postulates that the disagreement among an- notators in a language resource is a valuable source of knowledge, rather than noise that ought to be minimized or discarded (Plank, 2022; Basile et al., 2021b). Going one step further, the perspectivist approach aims at leveraging the disagreement in annotated data in order to model different points of view on the same phenomenon (Basile et al., 2021a). Applied to the study of natural language, this approach is particularly effective when the fo- cus phenomena belong to semantic and pragmatic areas (Abercrombie et al., 2022) such as undesir- able language detection, or irony and sarcasm. Although related, the interpretation of irony in- volves linguistic patterns, such as the reference to an opposite or secondary meaning, and prag- matic features (Karoui et al., 2017) which could make it possible to recognize the phenomenon for people with different social backgrounds. This differs from the perception of abusive language, proved to be highly affected by different subjectivi- ties (Akhtar et al., 2019). Thus, a …

Content truncated to 500 words