QCon at SemEval-2023 Task 10: Data Augmentation and Model Ensembling for Detection of Online Sexism

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1260–1270 July 13-14, 2023 ©2023 Association for Computational Linguistics QCon at SemEval-2023 Task 10: Data Augmentation and Model Ensembling for Detection of Online Sexism Weston Feely∗, Prabhakar Gupta∗, Manas Mohanty, Timothy Chon, Tuhin Kundu, Vijit Singh, Sandeep Atluri, Tanya Roosta, Viviane Ghaderi, Peter Schulam, Heba Elfardy Amazon {feelyw, prabhgup, mhmanas, timochon, tkundu, vijit, satluri, troosta, vsg, schulamp, helfardy} @amazon.com Abstract The web contains an abundance of user- generated content. While this content is use- ful for many applications, it poses many chal- lenges due to the presence of offensive, biased, and overall toxic language. In this work, we present a system that identifies and classifies sexist content at different levels of granularity. Using transformer-based models, we explore the value of data augmentation, use of ensem- ble methods, and leverage in-context learning using foundation models to tackle the task. We evaluate the different components of our sys- tem both quantitatively and qualitatively. Our best systems achieve an F1 score of 0.84 for the binary classification task – aiming to iden- tify whether a given content is sexist or not – and 0.64 and 0.47 for the two multi-class tasks that aim to identify the coarse and fine-grained types of sexism present in the given content respectively. 1 Introduction The web provides tremendous value, in part, be- cause of how easy it is to publish new content and for others to access it. This accessibility and open- ness, however, comes at a cost. Today’s web with its plethora of user-generated content is rife with toxic, offensive, and obscene language. Therefore, it has become essential to moderate this content at scale. Removing the toxic content is even more crit- ical for technologies that automatically index and surface web-content to end users, as well as gen- erative language models that utilize this content. To address the need for content moderation, the natural language processing (NLP) and machine learning communities have put considerable effort into the problem of automatically detecting toxic language. In this paper, we describe the system behind our team’s submission to the SemEval-2023 “Explain- able Detection of Online Sexism” shared task (Kirk ∗Equal contribution. et al., 2023). There are three tasks – Task A, B and C. Task A is a binary classification task and tasks B and C are multi-class classification tasks with 4 and 11 classes respectively. The current dominant approach to detecting toxic language is to fine-tune sequence to sequence transformer (Vaswani et al., 2017) models (Hanu and Unitary team, 2020; Mutanga et al., 2020; Kim et al., 2022; Markov et al., 2022). Our system builds on this thread of research, and extends it by applying methods to ensemble deep learning mod- els (Wenzel et al., 2020; Gontijo-Lopes et al., 2022; Wortsman et al., 2022). We ensemble transformers, where each member of the ensemble uses a differ- ent base architecture (i.e. BERT, RoBERTa, etc.), different training data, and different training hyper- parameter configurations. In addition, we explore …

Content truncated to 500 words