LT3 at SemEval-2020 Task 7: Comparing Feature-Based and Transformer-Based Approaches to Detect Funny Headlines

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1033–1040 Barcelona, Spain (Online), December 12, 2020. 1033 LT3 at SemEval-2020 Task 7: Comparing Feature-Based and Transformer-Based Approaches to Detect Funny Headlines Bram Vanroy, Sofie Labat, Olha Kaminska, Els Lefever and V´eronique Hoste LT3, Language and Translation Technology Team Department of Translation, Interpreting and Communication – Ghent University Groot-Brittanni¨elaan 45, 9000 Ghent, Belgium firstname.lastname@ugent.be Abstract This paper presents two different systems for the SemEval shared task 7 on Assessing Humor in Edited News Headlines, sub-task 1, where the aim was to estimate the intensity of humor generated in edited headlines. Our first system is a feature-based machine learning system that combines different types of information (e.g. word embeddings, string similarity, part-of-speech tags, perplexity scores, named entity recognition) in a Nu Support Vector Regressor (NuSVR). The second system is a deep learning-based approach that uses the pre-trained language model RoBERTa to learn latent features in the news headlines that are useful to predict the funniness of each headline. The latter system was also our final submission to the competition and is ranked seventh among the 49 participating teams, with a root-mean-square error (RMSE) of 0.5253. 1 Introduction Research in the field of computational humor can be divided into three distinct tasks: detecting, assessing and generating humor. Humor detection is performed by either distinguishing humorous items such as one-liners, jokes or texts from their non-humorous counterparts (Chen and Soo, 2018; Fan et al., 2020), or by ranking these items according to their respective funniness (Potash et al., 2017; Hossain et al., 2019). Moreover, researchers have tried to gain insights in the techniques and mechanisms that underlie humor production (Yang et al., 2015; Cattle and Ma, 2018; Hossain et al., 2019; West and Horvitz, 2019). Such insights can be used for automatic humor generation (Shahaf et al., 2015; He et al., 2019; Winters et al., 2019). Due to the subjective nature of humor and the deep conceptual common-sense knowledge that it requires, tasks involving humor remain challenging AI problems. Part of the challenge also lies in the scarcity of publicly available datasets. For the SemEval shared task on Assessing the Funninness of Edited News Headlines (Hossain et al., 2020), Hossain et al. (2019) introduce a novel dataset, named “Humicroedit”, that can be used for all three tasks described above (see Section 2). The shared task is split into two sub-tasks, namely (i) a regression task to predict the mean grade of all annotators for a given edited sentence, and (ii) a binary classification task to select the funnier of two edited headlines. We only participated in the regression sub-task. The remainder of this paper is structured as follows. In Section 2, we provide an overview of the dataset that was used for the competition. Section 3 introduces our two machine learning systems, namely (i) a feature-based regression model (NuSVR), and (ii) a state-of-the-art deep learning system. Next, in Section 4, our results for the two systems are compared and discussed. Finally, Section 5 …

Content truncated to 500 words