EndTimes at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with BERT and Ensembles

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1215–1220 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1215 EndTimes at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with BERT and Ensembles Chandan Kumar Pandey and Chirag Singh and Karan Mangla Samsung R&D Bangalore {chandan.p, c.singh, karan.mangla}@samsung.com Abstract This paper describes Humor-BERT, a set of BERT (Devlin et al., 2019) Large based mod- els that we used to solve the SemEval-2021 Task 7: Detecting and Rating Humor and Of- fense (Meaney et al., 2021). It presents pre and post processing techniques, variable thresh- old learning, meta learning and Ensemble ap- proach to solve various sub-tasks that were part of the challenge. We also present a com- parative analysis of various models we tried. Our method was ranked 4th in Humor Con- troversy Detection, 8th in Humor Detection, 19th in Average Offense Score prediction and 40th in Average Humor Score prediction glob- ally. F1 score obtained for Humor classifica- tion was 0.9655 and for Controversy detection it was 0.6261. Our user name on the leader board is ThisIstheEnd and team name is End- Times. 1 Introduction The purpose of this paper is to present different ap- proaches that we tried towards various sub-tasks in SemEval-2021 Task 7 Detecting and Rating Humor and Offense task (Meaney et al., 2021). It consists of following sub-tasks: • Task 1a: Classifying text based on whether they are humorous or not. • Task 1b: Predicting Humor rating score • Task 1c: If the text is classed as humorous, predict if the humor rating would be consid- ered controversial, i.e. the variance of the rating between annotators is higher than the median • Task 2a: predict how generally offensive a text is for users. This score was calculated regardless of whether the text is classed as humorous or offensive overall. We participated in all of the above sub-tasks. Task 1a and Task 1c are classification problems with F1 score as metric whereas, Task 1b and Task 2a are regression problems with RMS error values as scoring criteria. All the above tasks shared the same training, development and test set (Meaney et al., 2021). Training set consisted of 8000 text sentences, containing labels for all the subtasks. Size of development and test set was 1000 text sentences each. Humor, is an interesting linguistic challenge. It’s an abstract concept and depends a lot on audience and their interpretation of the jargon used to generate humor. Notion of humor changes from culture to culture, age group, gender and social status of target audience. It is a subjective and personal phenomenon. What’s humorous to one person could be non humorous or even offensive to the other. In this task, labels and ratings were collected from a balanced set of age groups from 18-70 and variety of genders, political stances and social status (Meaney et al., 2021). Humor detection and rating tasks have been the defined before but the task of detecting Controversy (Meaney et …

Content truncated to 500 words