DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1130–1134 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1130 DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods Bingyan Song Chunguang Pan Shengguang Wang Zhipeng Luo DeepBlue Technology (Shanghai) Co., Ltd {songby, panchg, wangshg, luozp}@deepblueai.com Abstract This paper describes the winning system for SemEval-2021 Task 7: Detecting and Rating Humor and Offense. Our strategy is stacking diverse pre-trained language models (PLMs) such as RoBERTa and ALBERT. We first per- form fine-tuning on these two PLMs with var- ious hyperparameters and different training strategies. Then a valid stacking mechanism is applied on top of the fine-tuned PLMs to get the final prediction. Experimental results on the dataset released by the organizer of the task show the validity of our method and we win first place and third place for subtask 2 and 1a. 1 Introduction Humor and offense detection continue to be chal- lenging AI problems since humor and offense in- volve in-depth world-knowledge, common sense, and the ability to perceive relationships across enti- ties and objects at various levels of understanding (Hossain et al., 2019). The recognition of humor and offense in the text has been receiving much attention (Zampieri et al., 2019; Hossain et al., 2020). Accordingly, SemEval-2021 Task 7, De- tecting and Rating Humor and Offense, which aims to automatically recognize humor in English jokes was held (Meaney et al., 2021). In this paper, we introduce our system for accom- plishing the above task by leveraging pre-trained models (PLMs). There are two main steps for our system, i) fine-tuning two kinds of PLMs, in- cluding ALBERT (Lan et al., 2019) and RoBERTa (Liu et al., 2019) with various hyperparameters and training strategies, achieving diverse models; ii) applying a validity stacking mechanism on top of these PLMs to do the final predictions. Our experimental results show that merging PLMs with different training strategies together can achieve great improvement which verifies the effectiveness of increasing model diversity. As a Tags No. of is humor Percentage train 4932 61.65% dev 632 63.20% test 615 61.50% Table 1: The number and percentage of humor samples in training set, validation set and test set respectively. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 train dev test 0~1 1~2 2~3 3~4 4~5 Figure 1: The distributions of offense rating for training set, validation set and test set. result, our system achieves the F1-score of 96.76% in subtask 1a and the RMSE of 41.2% in subtask 2, which ranks third and first among all the partici- pated teams respectively. 2 Background 2.1 Task Definition The “Detecting and Rating Humor and Offense” task, shared by SemEval-2021, consists of two sub- tasks. Subtask 1 includes three parts, a) A binary task to predict if the text would be considered hu- morous; b) A regression task to predict how hu- morous a text is if it is classed as humorous; c) A binary …

Content truncated to 500 words