PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 541–545 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis Elena Mikhalkova, Yuri Karyakin, Dmitry Grigoriev, Alexander Voronov, and Artem Leoznov Tyumen State University, Tyumen, Russia (e.v.mikhalkova, y.e.karyakin)@utmn.ru Abstract The paper describes our search for a univer- sal algorithm of detecting intentional lexical ambiguity in different forms of creative lan- guage. At SemEval-2018 Task 3, we used PunFields, the system of automatic analysis of English puns that we introduced at SemEval- 2017, to detect irony in tweets. Preliminary tests showed that it can reach the score of F1=0.596. However, at the competition, its re- sult was F1=0.549. 1 Introduction It requires no proof that usually language users can detect lexical ambiguity in figurative speech with- out classifying it into types like metaphor, pun, hu- mor, irony, sarcasm, etc. At the same time, such terminology is necessary in studyng these phe- nomena as well as solving problems of automatic classification. In our opinion, programs dealing with intentional lexical ambiguity should be more like natural language users, i.e. they should have one general mechanism of its detection optimized to a certain extent to different types of creative language. In Task 3 “Irony Detection in English Tweets” of SemEval-2018 (Van Hee et al., 2018), we attempted to employ PunFields, our previously introduced classifier of English puns (Mikhalkova and Karyakin, 2017b) 1, in the task of irony clas- sification in tweets as the constrained submission. I.e. PunFields trained only on the Gold dataset re- leased by the Organizers. As for the unconstrained submission, we used a bag-of-words model with an SVM classifier. The training dataset was enriched with tweets from other SemEval competitions. We will discuss it in a separate paragraph later in detail. The results of PunFields in this competition were above the chance value (cf.: Table 1) demon- 1https://github.com/evrog/PunFields Ac. Prec. Rec. F1 s. Con. 0.5765 0.4753 0.6495 0.5489 Unc. 0.6033 0.5000 0.5563 0.5266 Hom. 0.6782 0.7993 0.7337 0.7651 Het. 0.5747 0.7580 0.5940 0.6661 Table 1: Results of PunFields at SemEval 2018 Task 3 and SemEval 2017 Task 7. Ac. - acuracy, Prec. - precision, Rec. - recall, F1 s. - F1 score. SemEval- 2018: Con. - Constrained submission, Unc. - Uncon- strained submission. SemEval-2017: Hom. - homo- graphic puns, Het. - heterographic puns. strating higher scores of Recall and F1 score, but lower Acuracy and Precision compared to the en- riched bag-of-words (unconstrained submission). In analysis of puns, PunFields is more efficient reaching F1=0.765. In this paper we will briefly outline the state-of- the-art system and possible reasons for its compe- tition results. 2 Lexical Ambiguity in Irony and Puns We will demonstrate similarity between a pun and irony by the two following examples. Christmas shopping on 2 hours sleep is going to be fun. In this ironic tweet from the Gold dataset of the competition, the word fun is used simultaneously in the meaning “joy, …

Content truncated to 500 words