JUAGE at SemEval-2023 Task 10: Parameter Efficient Classification

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1195–1203 July 13-14, 2023 ©2023 Association for Computational Linguistics JUAGE at SemEval-2023 Task 10: Parameter Efficient Classification Jeffrey Sorensen Google Jigsaw sorenj@google.com Katerina Korre University of Bologna aikaterini.korre2@unibo.it John Pavlopoulos Athens University of Economics and Business annis@aueb.gr Katrin Tomanek Google Research katrintomanek @google.com Nithum Thain Google Research nthain@google.com Lucas Dixon Google Research ldixon@google.com Léo Laugier EPFL leo.laugier@epfl.ch Abstract Using pre-trained language models to imple- ment classifiers from small to modest amounts of training data is an area of active research. The ability of large language models to gener- alize from few-shot examples and to produce strong classifiers is extended using the engi- neering approach of parameter-efficient tuning. Using the Explainable Detection of Online Sex- ism (EDOS) training data and a small number of trainable weights to create a tuned prompt vector, a competitive model for this task was built, which was top-ranked in Subtask B. 1 Introduction Misogyny is a growing problem online: inflict- ing harm on the women who are targeted, making online spaces inaccessible and unwelcoming, and perpetuating social asymmetries and injustices. Au- tomated tools are now widely deployed to find, and assess sexist content at scale but most only give classifications for generic, high-level categories, with no further explanation. The Explainable De- tection of Online Sexism (EDOS) SemEval Task 10 was introduced to address this problem (Kirk et al., 2023), challenging English-language systems to flag what is more generally sexist content and also explaining why it is sexist. The JUAGE team participated by building one ensemble with BERT models for the most gen- eral subtask (A) and for the more detailed sub- tasks with taxonomies for sexism (B and C) sub- missions were made by tuning a PaLM derived model (Chung et al., 2022a), a 62-billion parame- ter large Pre-trained Language Models (PLM). The PaLM model variant was instruction-tuned (Chung et al., 2022b) and optimized for inference using quantization (Prato et al., 2020). While it is possible to fine-tune these large lan- guage models directly on user tasks, parameter- efficient tuning techniques keep the large language model weights fixed and enable sharing of re- sources across tasks and massively reduce the num- ber of parameters to be updated and stored. The smaller number of parameters that are updated also enable the rest of the model to act as a regularizer, allowing the model to learn from much smaller datasets (Mozes et al., 2023). Prompt engineering (Brown et al., 2020) is an- other popular way to use PLMs. However, given the constraints of the model input buffer, only a few training examples may be used (i.e., few-shot prompts for in-context learning). A recently pro- posed parameter-efficient alternative is to prepend pseudo-tokens to the input and then adjust the embedding weights of these tokens (“soft prompt learning”). This combines the resource sharing of a large, frozen model with a procedure for optimiz- ing weights over a training set. This technique has come to be known as prompt-tuning …

Content truncated to 500 words