RedwoodNLP at SemEval-2021 Task 7: Ensembled Pretrained and Lightweight Models for Humor Detection

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1209–1214 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1209 RedwoodNLP at SemEval-2021 Task 7: Ensembled Pretrained and Lightweight Models for Humor Detection Nathan A. Chi De Anza College Cupertino, CA chinathan@student.deanza.edu Ryan A. Chi Stanford University Stanford, CA ryanchi@stanford.edu Abstract An understanding of humor is an essential component of human-facing NLP systems. In this paper, we investigate several methods for detecting humor in short statements as part of Semeval-2021 Shared Task 7. For Task 1a, we apply an ensemble of fine-tuned pre-trained language models; for Tasks 1b, 1c, and 2a, we investigate various tree-based and linear machine learning models. Our final system achieves an F1-score of 0.9571 (ranked 24 / 58) on Task 1a, an RMSE of 0.5580 (ranked 18 / 50) on Task 1b, an F1-score of 0.5024 (ranked 26 / 36) on Task 1c, and an RMSE of 0.7229 (ranked 45 / 48) on Task 2a. 1 Introduction Humor detection is the process of identifying se- quences of text that are amusing—an important task, as such sequences are present in most chan- nels of communication. Although humor detection comes naturally to humans, it is difficult for artifi- cial systems to do the same. Part of the challenge is that it is debatable what constitutes humor; what one reader finds funny may be found utterly pro- saic by the next. The problem is only complicated when demographic factors come into play; now, the element of offense is also a factor. SemEval-2021 Shared Task 7 attempts to address some of these open problems (Meaney et al., 2021). Rather than definitively labeling text as humorous or not, Task 1a aims to determine whether the au- thor intended for the sentence to be humorous, Task 1b predicts its humor rating by the average user (its first moment), and Task 1c attends to whether the variance of its humor ratings (its second moment) exceeds the median. Task 2a, meanwhile, considers the text’s average offensiveness score, a metric that often correlates with whether the author meant the text to be humorous and—perhaps equally impor- tantly—affects whether the joke would be consid- ered acceptable. Overall, training models to per- form well on these tasks is of central importance to developing systems that are responsive to a wide range of input, whether in complete jest or meant to be taken at face value. 2 Dataset We train and validate our models on the SemEval- 2021 Task 7 training set (Table 6). Each English sentence is annotated for the following four labels, with continuous annotations labeled using a Likert scale from 1 to 5. # Description Label 1a Is the intention of this text Binary to be humorous? 1b How generally humorous is the Continuous text for the average user? 1c If the sentence is humorous, Binary is the the humor controversial? 1 2a How generally offensive is the text? Continuous Table 1: Annotations/subtasks with their descriptions. We submit separate models for …

Content truncated to 500 words