YNU-HPCC at SemEval-2020 Task 7: Using an Ensemble BiGRU Model to Evaluate the Humor of Edited News Titles

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 871–875 Barcelona, Spain (Online), December 12, 2020. 871 YNU-HPCC at SemEval-2020 Task 7: Using an ensemble BiGRU Model to evaluate the Humor of Edited News Titles Joseph J. Tomasulo, Jin Wang and Xuejie Zhang School of Information Science and Engineering Yunnan University Kunming, China Contact:{jtomasulo, wangjin, xjzhang}@ynu.edu.cn Abstract This paper describes a ensemble model designed for Semeval-2020 Task 7. The task is based on the Humicroedit dataset that is comprised of news titles and one-word substitutions designed to make them humorous. We use BERT, FastText, Elmo, and Word2Vec to encode these titles then pass them to a Bidirectional GRU with attention. Finally, we used XGBoost on the concatenation of the results of the different models to make predictions. 1 Introduction The Humicroedit dataset was created for research in computational humor (Hossain et al., 2019). The authors employed Amazon Mechanical Turk annotators to edit a single word of some fifteen thousand news headlines in order to make them funny. Five other annotators then graded each edited title on a scale of 0-3, with 3 being the funniest. Titles, despite being very short (the longest were around twenty words), convey a lot of information. Using one word edits is a way to make minimal changes for an expressed purpose. These two factors make the Humicroedit dataset a useful tool for studying computational humor. During the competition, the organizers published the Funlines dataset, which improved on Humi- croedit in terms of cost per title by gamifying the process: they provided cash rewards for the best annotators (Hossain et al., 2020b). The performance of annotators was measured on a mix of how funny their edits were and how well their grades tracked the overall average. Players were able to improve in both categories and the authors posit that this was due to the availability of live feedback. On average, the Funlines dataset is funnier than Humicroedit and there was better agreement on the grades. The Funlines dataset was made available to participants of the competition in January. There were two sub-tasks for SemEval Task 7: 1) for each headline and its edit, predict the mean of the grades assigned by the five annotators; 2) decide which of two title–edit pairs is funnier. We use a few popular pretrained language models in a BiGRU ensemble for the first task, and we use those results for the second task. The rest of the paper is organized as follows. Section 2 presents a brief review on humor assessment and some NLP tools. Section 3 presents the BiGRU ensemble model in detail. Section 4 contains some discussion of the results and section 5 is a short conlcusion. 2 Background Humor presents some challenges that other NLP tasks like entailment and analogy, for example, do not: it is highly contextual and subjective. Because of the latter, we may prefer to ask how funny something is or to whom is it funny, and when, over a question like “Is this …

Content truncated to 500 words