How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature

How Challenging is Sarcasm versus Irony Classification?: An Analysis From Human and Computational Perspectives Aditya Joshi1,2,3, Vaibhav Tripathi1, Pushpak Bhattacharyya1, Mark James Carman2 Meghna Singh1, Jaya Saraswati1, Rajita Shukla1 1Indian Institute of Technology Bombay, India, 2Monash University, Australia 3IITB-Monash Research Academy, India {adityaj, pb}@cse.iitb.ac.in, mark.carman@monash.edu mohan.meghnasingh@gmail.com, jaya.saraswati@gmail.com rajita.shukla38@gmail.com Abstract Sarcasm and irony, although similar, differ in that sarcasm has an impact on sentiment (be- cause it is used to ridicule a target) while irony does not. Past work treats the two interchange- ably. In this paper, we wish to validate if sarcasm versus irony classification is indeed a challenging task. To this end, we use a dataset of quotes from English literature, and conduct experiments from two perspectives: the human perspective and the computational perspective. For the former, we show that three human annotators have lower agreement for sarcasm versus irony as compared to sarcasm versus philosophy. Similarly, sarcasm versus irony classification performs with a lower F-score as compared to another classification task where labels are not related: sarcasm versus philoso- phy classification. Another key point that our paper makes is that features designed for sar- casm versus non-sarcasm classification do not work well for sarcasm versus irony classifica- tion. 1 Introduction Irony is a situation in which something which was in- tended to have a particular result has the opposite or a very different result1. On the other hand, sarcasm is a form of verbal irony that is intended to express con- tempt or ridicule. In other words, sarcasm has an ele- ment of ridicule and a target of ridicule, which is ab- sent in irony (Lee and Katz, 1998). For example, ‘He invented a new cure for a heart disease but later, died of the same disease himself’ is ironic but not sarcastic. However, ‘I didn’t make it big in Hollywood because I don’t write bad enough’ is sarcastic where the target of ridicule is Hollywood. Past work in sarcasm detection considers it as a sar- castic versus non-sarcastic classification (Kreuz and Caucci, 2007; Davidov et al., 2010; Gonz´alez-Ib´a˜nez et al., 2011). Alternately, Reyes et al. (2012) consider classification of irony/sarcasm versus humor. In many past approaches, sarcasm and irony are treated inter- changeably (Buschmeier et al., 2014; Joshi et al., 2015; 1Source: The Cambridge Dictionary Maynard and Greenwood, 2014). However, since sar- casm has a target that is being ridiculed, it is crucial that sarcasm be distinguished from mere irony. This is because when the target is identified, the sentiment of the target can be appropriately assigned. Owing to the two above reasons, sarcasm versus irony detection is a useful task. In this paper, we investigate sarcasm versus irony classification. To do so, we compare sarcasm ver- sus irony classification with sarcasm versus philosophy classification. In case of former, the two classes are similar (where sarcasm is hurtful/contemptuous). In case of sarcasm versus philosophy, the two classes are likely to be diverse. Thus, the goal of this paper is to to establish the challenging nature of sarcasm …

Content truncated to 500 words