Semi-Supervised Recognition of Sarcasm in Twitter and Amazon

Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Semi-Supervised Recognition of Sarcastic Sentences in Twitter and Amazon Dmitry Davidov ICNC The Hebrew University Jerusalem, Israel dmitry@alice.nc.huji.ac.il Oren Tsur Institute of Computer Science The Hebrew University Jerusalem, Israel oren@cs.huji.ac.il Ari Rappoport Institute of Computer Science The Hebrew University Jerusalem, Israel arir@cs.huji.ac.il Abstract Sarcasm is a form of speech act in which the speakers convey their message in an implicit way. The inherently ambiguous nature of sarcasm sometimes makes it hard even for humans to decide whether an ut- terance is sarcastic or not. Recognition of sarcasm can benefit many sentiment analy- sis NLP applications, such as review sum- marization, dialogue systems and review ranking systems. In this paper we experiment with semi- supervised sarcasm identification on two very different data sets: a collection of 5.9 million tweets collected from Twit- ter, and a collection of 66000 product re- views from Amazon. Using the Mechani- cal Turk we created a gold standard sam- ple in which each sentence was tagged by 3 annotators, obtaining F-scores of 0.78 on the product reviews dataset and 0.83 on the Twitter dataset. We discuss the dif- ferences between the datasets and how the algorithm uses them (e.g., for the Amazon dataset the algorithm makes use of struc- tured information). We also discuss the utility of Twitter #sarcasm hashtags for the task. 1 Introduction Sarcasm (also known as verbal irony) is a sophis- ticated form of speech act in which the speakers convey their message in an implicit way. One in- herent characteristic of the sarcastic speech act is that it is sometimes hard to recognize. The dif- ficulty in recognition of sarcasm causes misun- derstanding in everyday communication and poses problems to many NLP systems such as online review summarization systems, dialogue systems or brand monitoring systems due to the failure of state of the art sentiment analysis systems to detect sarcastic comments. In this paper we experiment with a semi-supervised framework for automatic identification of sarcastic sentences. One definition for sarcasm is: the activity of saying or writing the opposite of what you mean, or of speaking in a way intended to make someone else feel stupid or show them that you are angry (Macmillan English Dictionary (2007)). Using the former definition, sarcastic utterances appear in many forms (Brown, 1980; Gibbs and O’Brien, 1991). It is best to present a number of examples which show different facets of the phenomenon, followed by a brief review of different aspects of the sarcastic use. The sentences are all taken from our experimental data sets: 1. “thank you Janet Jackson for yet another year of Super Bowl classic rock!” (Twitter) 2. “He’s with his other woman: XBox 360. It’s 4:30 fool. Sure I can sleep through the gun- fire” (Twitter) 3. “Wow GPRS data speeds are blazing fast.” (Twitter) 4. “[I] Love The Cover” (book, amazon) 5. “Defective by design” (music player, ama- zon) …

Content truncated to 500 words