Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1049–1054 Barcelona, Spain (Online), December 12, 2020. 1049 Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating J. A. Meaney Steven R. Wilson Walid Magdy Institute for Language, Cognition and Computation School of Informatics University of Edinburgh {jameaney, steven.wilson}@ed.ac.uk, wmagdy@inf.ed.ac.uk Abstract The use of pre-trained language models such as BERT and ULMFiT has become increasingly popular in shared tasks, due to their powerful language modelling capabilities. Our entry to SemEval uses ERNIE 2.0, a language model which is pre-trained on a large number of tasks to enrich the semantic and syntactic information learned. ERNIE’s knowledge masking pre- training task is a unique method for learning about named entities, and we hypothesise that it may be of use in a dataset which is built on news headlines and which contains many named entities. We optimize the hyperparameters in a regression and classification model and find that the hyperparameters we selected helped to make bigger gains in the classification model than the regression model. 1 Introduction Verbal humor uses a variety of linguistic features, such as synonymy, wordplay, and phonological sim- iliarities, as well non-linguistic features like world knowledge, to produce a comic effect. That such a broad set of skills are required to understand humor, has led several researchers to deem that computa- tional humor is an AI-complete problem (Stock and Strapparava, 2006; Binsted et al., 2006). There is a relatively longstanding body of research into humor detection in a limited domain, such as knock-knock jokes (Taylor and Mazlack, 2004), one-liners (Mihalcea and Strapparava, 2006) and humorous news ar- ticles from the satirical news publication The Onion (Mihalcea and Pulman, 2007). However, the use of shared tasks has attracted more attention and interest in the field since 2017. While previous challenges have focused on collecting Twitter data (Potash et al., 2017; Castro et al., 2018), SemEval 2020 (Hossain et al., 2020) took an original approach and generated the data by collecting news headlines and then asking annotators to edit one word in the headline to make it humorous (Hossain et al., 2019). These headlines emulate those of The Onion. The edits shown below indicate the location of the substitution and the word to be inserted. Table 1: Example of Funlines Headline Type Text Edit Score Original Mitch McConnell thinks tax reform will take longer than Trump claimed Edit 1 Mitch McConnell thinks tax reform will take longer than Trump <claimed/> haircut 2.8 Edit 2 Mitch McConnell thinks tax <reform/>will take longer than Trump claimed return 1.6 The edited headlines were then rated for humor by subsequent annotators. Sub-task A was to predict the mean funniness score of the edited headline. In sub-task B, the systems saw two edits of the same headline, and predicted which one had achieved the higher mean funniness score. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/. 1050 2 Previous Work Excluding work …

Content truncated to 500 words