SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802 - 814 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic Ibrahim Abu Farha1, Silviu Vlad Oprea1, Steven R. Wilson1,3, Walid Magdy1,2 1 School of Informatics, The University of Edinburgh, Edinburgh, UK 2 The Alan Turing Institute, London, UK 3 Oakland University, Rochester, MI, USA {i.abufarha,silviu.oprea,steven.wilson,wmagdy}@ed.ac.uk Abstract iSarcasmEval is the first shared task to target intended sarcasm detection: the data for this task was provided and labelled by the authors of the texts themselves. Such an approach min- imises the downfalls of other methods to collect sarcasm data, which rely on distant supervi- sion or third-party annotations. The shared task contains two languages, English and Arabic, and three subtasks: sarcasm detection, sarcasm category classification, and pairwise sarcasm identification given a sarcastic sentence and its non-sarcastic rephrase. The task received submissions from 60 different teams, with the sarcasm detection task being the most popu- lar. Most of the participating teams utilised pre-trained language models. In this paper, we provide an overview of the task, data, and par- ticipating teams. 1 Introduction Sarcasm is a form of verbal irony that occurs when there is a discrepancy between the literal and in- tended meanings of an utterance. Through this discrepancy, the speaker expresses their position towards a prior proposition, often in the form of surface contempt or derogation (Wilson, 2006). Sarcasm is present on the social web and, due to its nature, it can be disruptive of computational systems that harness this data to perform tasks such as sentiment analysis, opinion mining, author pro- filing, and harassment detection (Liu, 2012; Rosen- thal et al., 2014; Maynard and Greenwood, 2014; Van Hee et al., 2018). In the context of SemEval, in particular, Rosenthal et al. (2014) show a signifi- cant drop in sentiment polarity classification perfor- mance when processing sarcastic tweets, compared to non-sarcastic ones. Such computational systems are widely deployed in industry, driving marketing, administration, and investment decisions (Medhat et al., 2014). In the context of Arabic, Abu Farha and Magdy (2021) show the effect of sarcasm on Arabic sentiment analysis systems, where the per- formance dropped significantly for the sarcastic tweets. As such, it is imperative to devise models for sarcasm detection. Such models are usually built in a supervised learning paradigm, relying on a dataset of texts labelled as either sarcastic or non-sarcastic. Two methods have typically been used to label texts for sarcasm: distant supervision (Ptáˇcek et al., 2014; Khodak et al., 2018; Barbieri et al., 2014), where texts are considered sarcastic if they meet prede- fined criteria, such as including the tag #sarcasm; or manual labelling (Filatova, 2012; Riloff et al., 2013a; Abercrombie and Hovy, 2016), where texts are presented to human annotators. However, as argued by Oprea and Magdy (2020a), both meth- ods could produce noisy labels, in terms of both false positives, and false negatives. For instance, when human annotators label texts, they are limited …

Content truncated to 500 words