SemEval-2018 Task 3: Irony Detection in English Tweets

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 39–50 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 3: Irony Detection in English Tweets Cynthia Van Hee, Els Lefever and V´eronique Hoste LT3 Language and Translation Technology Team Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent firstname.lastname@ugent.be Abstract This paper presents the first shared task on irony detection: given a tweet, automatic nat- ural language processing systems should de- termine whether the tweet is ironic (Task A) and which type of irony (if any) is expressed (Task B). The ironic tweets were collected us- ing irony-related hashtags (i.e. #irony, #sar- casm, #not) and were subsequently manually annotated to minimise the amount of noise in the corpus. Prior to distributing the data, hash- tags that were used to collect the tweets were removed from the corpus. For both tasks, a training corpus of 3,834 tweets was provided, as well as a test set containing 784 tweets. Our shared tasks received submissions from 43 teams for the binary classification Task A and from 31 teams for the multiclass Task B. The highest classification scores obtained for both subtasks are respectively F1= 0.71 and F1= 0.51 and demonstrate that fine-grained irony classification is much more challenging than binary irony detection. 1 Introduction The development of the social web has stimulated the use of figurative and creative language, includ- ing irony, in public (Ghosh et al., 2015). From a philosophical/psychological perspective, discern- ing the mechanisms that underlie ironic speech im- proves our understanding of human reasoning and communication, and more and more, this interest in understanding irony also emerges in the ma- chine learning community (Wallace, 2015). Al- though an unanimous definition of irony is still lacking in the literature, it is often identified as a trope whose actual meaning differs from what is literally enunciated. Due to its nature, irony has important implications for natural language pro- cessing (NLP) tasks, which aim to understand and produce human language. In fact, automatic irony detection has a large potential for various appli- cations in the domain of text mining, especially those that require semantic analysis, such as au- thor profiling, detecting online harassment, and, maybe the most well-known example, sentiment analysis. Due to its importance in industry, sentiment analysis research is abundant and significant progress has been made in the field (e.g. in the con- text of SemEval (Rosenthal et al., 2017)). How- ever, the SemEval-2014 shared task Sentiment Analysis in Twitter (Rosenthal et al., 2014) demon- strated the impact of irony on automatic senti- ment classification by including a test set of ironic tweets. The results revealed that, while senti- ment classification performance on regular tweets reached up to F1= 0.71, scores on the ironic tweets varied between F1= 0.29 and F1= 0.57. In fact, it has been demonstrated that several applica- tions struggle to maintain high performance when applied to ironic text (e.g. Liu, 2012; Maynard and Greenwood, 2014; Ghosh and Veale, 2016). Like other types of figurative …

Content truncated to 500 words