Large Dataset and Language Model Fun-Tuning for Humor Recognition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4027–4032 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4027 Large Dataset and Language Model Fun-Tuning for Humor Recognition Vladislav Blinov1,3, Valeriia Bolotova-Baranova1,3, and Pavel Braslavski1,2 1Ural Federal University, Yekaterinburg, Russia 2Higher School of Economics, Saint Petersburg, Russia 3Tinkoff.ru {vladislav.blinov,pavel.braslavsky}@urfu.ru, lurunchik@gmail.com Abstract The task of humor recognition has attracted a lot of attention recently due to the urge to pro- cess large amounts of user-generated texts and rise of conversational agents. We collected a dataset of jokes and funny dialogues in Rus- sian from various online resources and com- plemented them carefully with unfunny texts with similar lexical properties. The dataset comprises of more than 300,000 short texts, which is significantly larger than any previous humor-related corpus. Manual annotation of about 2,000 items proved the reliability of the corpus construction approach. Further, we ap- plied language model fine-tuning for text clas- sification and obtained an F1 score of 0.91 on test set, which constitutes a considerable gain over baseline methods. The dataset is freely available for research community. 1 Introduction Humor is an important element of everyday hu- man communication (Martin, 2007). With a rapid development of conversational systems and NLP applications for social media content, the task of automatic recognition of humor and other types of figurative language has gained a lot of atten- tion (Nijholt et al., 2017). Standard publicly avail- able datasets significantly contribute to steady and measurable progress in solving NLP tasks. To the date, there are several humor-related datasets, but the majority of them contain English texts only, are relatively small, and focus predominantly on puns, thus don’t reflect a wide variety of humor- ous content. In this work we describe the creation of a large dataset of funny short texts in Russian. We started with an existing dataset and more than tripled it in size. The texts were automatically collected from various online sources to ensure their diver- sity and representativeness. A separate task was the compilation of a contrasting corpus of unfunny texts in such a way that their distinguishing char- acteristic was absence of humor, and not their lex- ical properties and style. The dataset comprises of more than 300,000 short texts in total, about half of them being funny. Manual annotation of 1,877 examples confirmed the validity of the automatic approach and formed a golden test set. We implemented a humor detection method based on the universal language model fine- tuning. Unlike most approaches to humor recog- nition described in the literature, this method nei- ther draws upon an existing theory of humor, nor makes explicit assumptions about the structure and ‘mechanics’ of jokes; it needs no feature engineer- ing and is purely data-driven. This approach is jus- tified in the case of a large heterogeneous collec- tion. Evaluation of the trained model on several test collections of Russian jokes shows that it has not been overfitted and generalizes well. The compiled dataset …

Content truncated to 500 words