Humor@IITK at SemEval-2021 Task 7: Large Language Models for Quantifying Humor and Offensiveness

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 290–296 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 290 Humor@IITK at SemEval-2021 Task 7: Large Language Models for Quantifying Humor and Offensiveness Aishwarya Gupta∗, Avik Pal∗, Bholeshwar Khurana∗, Lakshay Tyagi∗, Ashutosh Modi Indian Institute of Technology Kanpur (IIT Kanpur) {aishwaryag20, avikpal, bholek, lakshayt}@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract Humor and Offense are highly subjective due to multiple word senses, cultural knowledge, and pragmatic competence. Hence, accurately detecting humorous and offensive texts has several compelling use cases in Recommenda- tion Systems and Personalized Content Moder- ation. However, due to the lack of an extensive labeled dataset, most prior works in this do- main haven’t explored large neural models for subjective humor understanding. This paper explores whether large neural models and their ensembles can capture the intricacies associ- ated with humor/offense detection and rating. Our experiments on the SemEval-2021 Task 7: HaHackathon show that we can develop rea- sonable humor and offense detection systems with such models. Our models are ranked third in subtask 1b and consistently ranked around the top 33% of the leaderboard for the remain- ing subtasks. 1 Introduction Like most figurative languages, humor/offense pose interesting linguistic challenges to Natural Language Processing due to its emphasis on mul- tiple word senses, cultural knowledge, sarcasm, and pragmatic competence. A joke’s perception is highly subjective, and age, gender, and socioeco- nomic status extensively influence it. Prior humor detection/rating challenges treated humor as an ob- jective concept. SemEval 2021 Task 7 (Meaney et al., 2021) is the first humor detection challenge that incorporates the subjectivity associated with humor and offense across different demographic groups. Users from varied age groups and genders annotated the data with the text’s humor and have provided an associated score for the same. It is also quite a generic phenomenon that a text might be ∗ Authors contributed equally to the work. Names is alphabetical order. humorous to one and normal/offensive to another. Rarely has it been noticed that the same content is globally accepted as witty. To the best of our knowledge, Meaney et al. (2021) is the first initia- tive towards annotating the underlying humor as controversial or not. Understanding whether a text is humorous and/or offensive will aid downstream tasks, such as personalized content moderation, rec- ommendation systems, and flagging offensive con- tent. Large Language Models (LLMs) have recently emerged as the SOTA for various Natural Lan- guage Understanding Tasks (Lewis et al., 2019; Raffel et al., 2019; Conneau et al., 2019; Zhang et al., 2020). However, typical day-to-day texts, where these models have shown state of the art performance, are less ambiguous than texts hav- ing puns/jokes. Training and evaluating LLMs in the context of highly ambiguous/subjective English texts would serve as an excellent benchmark to fig- ure out the current shortcomings of these models. This paper studies various large language models – BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), ERNIE-2.0 (Sun et al., …

Content truncated to 500 words