LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105–119 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 105 SemEval-2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense J.A. Meaney1, Steven R. Wilson1, Luis Chiruzzo2, Adam Lopez1, 3, Walid Magdy1,4 1 School of Informatics, The University of Edinburgh, Edinburgh, UK 2 Universidad de la Rep´ublica, Uruguay 3 Rasa 4 The Alan Turing Institute, London, UK {jameaney, steven.wilson}@ed.ac.uk {alopez, wmagdy}@inf.ed.ac.uk luischir@fing.edu.uy Abstract SemEval 2021 Task 7, HaHackathon, was the first shared task to combine the previously sep- arate domains of humor detection and offense detection. We collected 10,000 texts from Twitter and the Kaggle Short Jokes dataset, and had each annotated for humor and offense by 20 annotators aged 18-70. Our subtasks were binary humor detection, prediction of humor and offense ratings, and a novel con- troversy task: to predict if the variance in the humor ratings was higher than a specific threshold. The subtasks attracted 36-58 sub- missions, with most of the participants choos- ing to use pre-trained language models. Many of the highest performing teams also imple- mented additional optimization techniques, in- cluding task-adaptive training and adversarial training. The results suggest that the partic- ipating systems are well suited to humor de- tection, but that humor controversy is a more challenging task. We discuss which models excel in this task, which auxiliary techniques boost their performance, and analyze the er- rors which were not captured by the best sys- tems. 1 Introduction Humor is a key component of many forms of com- munication, and so it is commanding an increasing amount of attention in the natural language process- ing (NLP) community (Attardo, 2008; Taylor and Attardo, 2017; Amin and Burghardt, 2020). How- ever, like much of figurative language processing, humor detection requires a different perspective on several traditional NLP tasks. For example, the problem of reducing lexical or syntactic ambigu- ity differs when ambiguity is key to some humor mechanisms. Tackling these challenges has the po- tential to improve many downstream applications, such as content moderation and human-computer interaction (Rayz, 2017). However, humor is a subjective phenomenon, which evokes varying degrees of funniness in its audience, while also provoking other reactions such as offense, in certain listeners. The perception of humor is known to vary along the lines of age, gender, personality and other factors (Ruch, 2010; Kuipers, 2015; Hofmann et al., 2020). That hu- mor can also evoke offense may be partly due to differences in acceptability judgements across de- mographic groups, and may also be in part due the use of humor to mask hateful or offensive content (Sue and Golash-Boza, 2013). Lockyer and Picker- ing (2005) expand on this by highlighting that it is common for societies to explore the link between humor and offense, free speech and respect. HaHackathon is the first shared task to combine humor and offense detection, based on ratings from a wide variety of demographic groups. Task partic- ipants were asked to detect if a …

Content truncated to 500 words