iSarcasm: A Dataset of Intended Sarcasm

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1279 iSarcasm: A Dataset of Intended Sarcasm Silviu Vlad Oprea School of Informatics University of Edinburgh Edinburgh, United Kingdom silviu.oprea@ed.ac.uk Walid Magdy School of Informatics University of Edinburgh Edinburgh, United Kingdom wmagdy@inf.ed.ac.uk Abstract We consider the distinction between intended and perceived sarcasm in the context of tex- tual sarcasm detection. The former occurs when an utterance is sarcastic from the per- spective of its author, while the latter occurs when the utterance is interpreted as sarcastic by the audience. We show the limitations of previous labelling methods in capturing in- tended sarcasm and introduce the iSarcasm dataset of tweets labeled for sarcasm directly by their authors. Examining the state-of-the- art sarcasm detection models on our dataset showed low performance compared to previ- ously studied datasets, which indicates that these datasets might be biased or obvious and sarcasm could be a phenomenon under-studied computationally thus far. By providing the iS- arcasm dataset, we aim to encourage future NLP research to develop methods for detect- ing sarcasm in text as intended by the authors of the text, not as labeled under assumptions that we demonstrate to be sub-optimal. 1 Introduction Sarcasm is a form of irony that occurs when there is some discrepancy between the literal and intended meanings of an utterance. This discrepancy is used to express dissociation towards a previous propo- sition, often in the form of contempt or deroga- tion (Wilson, 2006). Sarcasm is omnipresent in social media text and can be highly disruptive of systems that harness this data for sentiment and emotion analysis (Maynard and Greenwood, 2014). It is therefore imperative to devise models for sar- casm detection. The effectiveness of such models depends on the availability and quality of labelled data used for training. Collecting such data is chal- lenging due to the subjective nature of sarcasm. For instance, Dress et al. (2008) notice a lack of consistence in how sarcasm is used by people of different socio-cultural backgrounds. As a result, an utterance intended sarcastic by its author might not be perceived as such by audiences of different backgrounds (Rockwell and Theriot, 2001; Oprea and Magdy, 2020). There are two methods used so far to label texts for sarcasm: distant supervision, where texts are considered sarcastic if they meet predefined criteria, such as including specific hashtags; and manual labelling by human annotators. We believe both methods are sub-optimal for capturing the sarcastic intention of the authors of the texts. As a result, existing models trained on such datasets might be optimized to capture the noise induced by these labelling methods. In this paper, we present the iSarcasm dataset of tweets labelled for sarcasm by their authors. To our knowledge, this is the first attempt to create noise- free examples of intended sarcasm. In a survey, we asked Twitter users to provide both sarcastic and non-sarcastic tweets that they had posted in …

Content truncated to 500 words