Reactive Supervision: A New Method for Collecting Sarcasm Data

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2553–2559, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2553 Reactive Supervision: A New Method for Collecting Sarcasm Data Boaz Shmueli1,2,3,∗, Lun-Wei Ku2 and Soumya Ray3 1Social Networks and Human-Centered Computing, Taiwan International Graduate Program 2Institute of Information Science, Academia Sinica 3Institute of Service Science, National Tsing Hua University Abstract Sarcasm detection is an important task in af- fective computing, requiring large amounts of labeled data. We introduce reactive supervi- sion, a novel data collection method that uti- lizes the dynamics of online conversations to overcome the limitations of existing data col- lection techniques. We use the new method to create and release a first-of-its-kind large dataset of tweets with sarcasm perspective la- bels and new contextual features. The dataset is expected to advance sarcasm detection re- search. Our method can be adapted to other affective computing domains, thus opening up new research opportunities. 1 Introduction Sarcasm is ubiquitous in human conversations. As a form of insincere speech, the intent behind a sarcastic utterance is integral to its meaning. Per- ceiving a sarcastic utterance as genuine will often result in a complete reversal of the intended mean- ing, and vice versa (Gibbs, 1986). It is therefore crucial for affective computing systems and tasks, such as sentiment analysis and dialogue systems, to automatically detect sarcasm from the perspective of the author as well as the reader in order to avoid misunderstandings. Oprea and Magdy (2019) re- cently pioneered the study of intended sarcasm (by the author) vs. perceived sarcasm (by the reader) in the context of sarcasm detection tasks. The training of models for these tasks requires large amounts of labeled sarcasm data, with Twitter becoming a ma- jor source due to its popularity as a social network as well as the huge amounts of conversational text its users generate. Previous works describe three methods for collecting sarcasm data: distant super- vision, manual annotation, and manual collection. Distant supervision automatically collects “in- the-wild” sarcastic tweets by leveraging author- ∗Corresponding author: shmueli@iis.sinica.edu.tw generated labels such as the #sarcasm hashtag (Davidov et al., 2010; Pt´aˇcek et al., 2014). This method generates large amounts of data at low cost, but labels are often noisy and biased (Bamman and Smith, 2015). To improve quality, manual annotation asks hu- mans to label given tweets as sarcastic or not. Since finding sarcasm in a large corpus is “a needle-in-a- haystack problem” (Liebrecht et al., 2013), manual annotation can be combined with distant supervi- sion (Riloff et al., 2013). Still, low inter-annotator reliability is often reported (Swanson et al., 2014), resulting not only from the subjective nature of sar- casm but also the lack of cultural context (Joshi et al., 2016). Moreover, neither method collects both sarcasm perspectives: distant supervision col- lects intended sarcasm, while manual annotation can only collect perceived sarcasm. Lastly, in manual collection, humans are asked to gather and report sarcastic texts, either their own (Oprea and Magdy, 2020) or by others …

Content truncated to 500 words