A Large Self-Annotated Corpus for Sarcasm

A Large Self-Annotated Corpus for Sarcasm Mikhail Khodak, Nikunj Saunshi, Kiran Vodrahalli Computer Science Department, Princeton University 35 Olden St., Princeton, New Jersey 08540 {mkhodak,nsaunshi,knv}@cs.princeton.edu Abstract We introduce the Self-Annotated Reddit Corpus (SARC), a large corpus for sarcasm research and for training and evaluating systems for sarcasm detection. The corpus has 1.3 million sarcastic statements — 10 times more than any previous dataset — and many times more instances of non-sarcastic statements, allowing for learning in both balanced and unbalanced label regimes. Each statement is furthermore self-annotated — sarcasm is labeled by the author, not an independent annotator — and provided with user, topic, and conversation context. We evaluate the corpus for accuracy, construct benchmarks for sarcasm detection, and evaluate baseline methods. Keywords: sarcasm, classification, conversation 1 Introduction Sarcasm detection is an important component in many nat- ural language processing (NLP) systems, directly relevant to natural language understanding, dialogue systems, and text mining. However, detecting sarcasm is difficult be- cause it occurs infrequently and is difficult for even humans to discern (Wallace et al., 2014). Despite these proper- ties, existing datasets either have balanced labels — data with roughly the same number of examples of each label (Gonz´alez-Ib´anez et al., 2011; Bamman and Smith, 2015; Joshi et al., 2015; Amir et al., 2016; Oraby et al., 2016) — or use humans to annotate sarcastic statements (Riloff et al., 2013; Swanson et al., 2014; Wallace et al., 2015). In this work, we make available the first corpus1 for sar- casm detection that has both unbalanced and self-annotated labels and does not consist of short text snippets from Twit- ter2. With more than a million examples of sarcastic state- ments, each provided with author, topic, and contex infor- mation, the dataset exceeds all previous sarcasm corpora by an order of magnitude in size. This is possible due to the comment structure of the social media site Reddit3 as well as its frequently-used and standardized annotation for sarcasm. Following a discussion of corpus construction and rele- vant statistics in Section 3, we discuss the quality of this dataset compared to alternative sources in Section 4, man- ually evaluating our corpus for noise. Then in Section 5 we use our dataset to construct suitable benchmarks for sar- casm detection systems and examine the performance of simple baseline methods and human evaluators on these subsets. 2 Related Work Since our main contribution is a corpus and not a method for sarcasm detection, we point the reader to a recent sur- vey by Joshi et al. (2016) that discusses many interesting efforts in this area. Note that many of the works the authors mention will be discussed by us in this section, with many 1http://nlp.cs.princeton.edu/SARC/ 2https://www.twitter.com 3https://www.reddit.com papers using their own datasets and illustrating the need for common evaluation baselines. Sarcasm datasets can largely be distinguished by the sources used to get sarcastic and non-sarcastic statements, the amount of human annotation, and whether the dataset is balanced or unbalanced. Reddit has been used before, notably by …

Content truncated to 500 words