YoungSheldon at SemEval-2021 Task 7: Fine-tuning Is All You Need

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1146–1152 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1146 YoungSheldon at SemEval-2021 Task 7: Fine-tuning Is All You Need Mayukh Sharma, Ilanthenral Kandasamy, W.B. Vasantha School of Computer Science and Engineering Vellore Institute of Technology Vellore, Tamil Nadu, India 04mayukh@gmail.com, ilanthenral.k@vit.ac.in, vasantha.wb@vit.ac.in Abstract In this paper, we describe our system used for SemEval 2021 Task 7: HaHackathon: Detect- ing and Rating Humor and Offense. We used a simple fine-tuning approach using different Pre-trained Language Models (PLMs) to eval- uate their performance for humor and offense detection. For regression tasks, we averaged the scores of different models leading to bet- ter performance than the original models. We participated in all SubTasks. Our best perform- ing system was ranked 4 in SubTask 1-b, 8 in SubTask 1-c, 12 in SubTask 2, and performed well in SubTask 1-a. We further show compre- hensive results using different pre-trained lan- guage models which will help as baselines for future work. 1 Introduction Humor is an intelligent form of communication with the capability of providing amusement and provoking laughter (Chen and Soo, 2018). It helps in bridging the gap between various languages, cul- tures, and demographics. Humor is a very sub- jective phenomenon. It can have different intensi- ties, and people may find some jokes funnier than others. In certain situations, some jokes may be offensive to a certain group of people. All these characteristics of humor pose an interesting linguis- tic challenge to NLP systems. SemEval 2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense (Meaney et al., 2021) aims to draw attention to these challenges in humor detection. The task provides a dataset of humorous content annotated using people representing different age groups, gender, political stances, and income levels. The content of the provided dataset was in English. Participating in all SubTasks, we propose a fine- tuning based approach on pre-trained language models. Pre-trained Language Models learn syn- tactic and semantic representations by training on large amounts of unsupervised data. Recently there has been a lot of interest in PLMs. Re- searchers have come up with different pre-training methods using Auto Encoding(AE) and Auto- Regressive(AR) language modeling techniques. Of- ten these pre-trained models contain millions of pa- rameters and are computationally expensive. Fine- tuning different models may lead to different re- sults on downstream tasks. This makes the choice of PLM an important factor. We present a com- parative study of different PLM models and their performance in all SubTasks of SemEval 2021 Task 7: HaHackathon. Our proposed fine-tuning approach for each PLM made use of a single layer of one neuron stacked on the PLM features. We performed exper- iments using BERT(Devlin et al., 2019), ELEC- TRA(Clark et al., 2020), RoBERTa(Liu et al., 2019), XLNet(Yang et al., 2019), MPNet(Song et al., 2020), and ALBERT(Lan et al., 2020). For regression tasks, we also used averaging technique, which we describe later in the paper. Our model performed well in …

Content truncated to 500 words