LRG at SemEval-2020 Task 7: Assessing the Ability of BERT and Derivative Models to Perform Short-Edits Based Humor Grading

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 858–864 Barcelona, Spain (Online), December 12, 2020. 858 LRG at SemEval-2020 Task 7: Assessing the Ability of BERT and Derivative Models to Perform Short-Edits based Humor Grading Siddhant Mahurkar Department of Computer Science & Engineering Vellore Institute of Technology, Vellore, India siddhant.pravin2017@vitstudent.ac.in Rajaswa Patil Department of Electrical & Electronics Engineering BITS Pilani K. K. Birla Goa Campus, India f20170334@goa.bits-pilani.ac.in Abstract In this paper, we assess the ability of BERT and its derivative models (RoBERTa, DistilBERT, and ALBERT) for short-edits based humor grading. We test these models for humor grading and classification tasks on the Humicroedit and the FunLines dataset. We perform extensive experiments with these models to test their language modeling and generalization abilities via zero-shot inference and cross-dataset inference based approaches. Further, we also inspect the role of self-attention layers in humor-grading by performing a qualitative analysis over the self-attention weights from the final layer of the trained BERT model. Our experiments show that all the pre-trained BERT derivative models show significant generalization capabilities for humor-grading related tasks. 1 Introduction Humor is a communicative ability that produces a sense of laughter by imparting an amusing tone in conversations. If we can teach our models how to generate, detect, and grade humor in language, it would be a big step ahead in Natural Language Processing. Nearly all the existing datasets for humor-related tasks such as ”Pun of the Day” (Yang et al., 2015) and ”16000 One Liners” (Mihalcea and Strapparava, 2005) focus on humor-detection in a single piece of text as a binary classification task. Previous tasks on humor-grading consist of datasets comprising of originally humorous texts (Potash et al., 2017; Chiruzzo et al., 2019). SemEval-2020 Task-7 (Hossain et al., 2020a) focuses on short-edits based humor-grading. The dataset used for this shared-task was introduced by Hossain et al. (2019). It consists of short-edits applied to a piece of text (news-headlines), which changes it from non-funny to potentially funny. This shared-task consists of two independent sub-tasks based on this dataset. The first sub-task comprises of a regression task, where given the original and the edited news-headline, one must predict the funniness of the edited headline (humor-grading) on a scale of [0,3], where ’0’ conveys ’least funny’ and ’3’ conveys ’most funny’. The second sub-task comprises of a classification task, where given two edited versions of the same news-headline, the model must predict the funnier one. In this work, we propose using BERT (Devlin et al., 2019) and its derivative models (collectively referred to as ”BERT models” further throughout the paper) for short-edits based humor-grading. We mainly follow two approaches: In the first one we use pre-trained BERT models with a final regression layer to get a humor-grade for each edited news-headline, and for the second one we use masked-word language-modeling to fine-tune the pre-trained BERT models on the task dataset, which are further trained for the humor-grade regression task. We compare the results of these two approaches for each sub-task. We …

Content truncated to 500 words