Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1329–1334 July 13-14, 2023 ©2023 Association for Computational Linguistics Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction Peyman Hosseini1, Mehran Hosseini2, Sana Sabah Al-Azzawi3, Marcus Liwicki3, Ignacio Castro1, and Matthew Purver1,4 1Queen Mary University of London, London, United Kingdom {s.hosseini, m.purver, i.castro}@qmul.ac.uk 2King’s College London, London, United Kingdom mehran.hosseini@kcl.ac.uk 3Luleå University of Technology, Luleå, Sweden {sana.al-azzawi, marcus.liwicki}@ltu.se 4Jožef Stefan Institute, Ljubljana, Slovenia Abstract We study the influence of different activation functions in the output layer of deep neural network models for soft and hard label predic- tion in the learning with disagreement task. In this task, the goal is to quantify the amount of disagreement via predicting soft labels. To predict the soft labels, we use BERT-based pre- processors and encoders and vary the activation function used in the output layer, while keeping other parameters constant. The soft labels are then used for the hard label prediction. The acti- vation functions considered are sigmoid as well as a step-function that is added to the model post-training and a sinusoidal activation func- tion, which is introduced for the first time in this paper. 1 Introduction The nuances present in natural languages, such as contextual meaning or subjective interpretation of expressions are often discounted in natural lan- guage processing. In subjective tasks, such as sen- timent analysis, offensive or abusive language de- tection, and misogyny detection, the assumption that a gold label always exists has proven to be an idealization (Uma et al., 2021). Therefore, a growing body of research has been dedicated to analyzing the disagreement from la- bels provided by multiple annotators. Some of the notable works on this topic are (Uma et al., 2021; Fornaciari et al., 2021; Uma et al., 2020; Kenyon- Dean et al., 2018). In task 11 of SemEval 2023 on Learning with Disagreements Le-Wi-Di, 2nd edition (Leonardellli et al., 2023), four datasets, including three English datasets HS-Brexit (Akhtar et al., 2021), ConvAbuse (Cercas Curry et al., 2021), and MD-Agreement (Leonardelli et al., 2021), and one Arabic dataset, ArMIS (Almanea and Poesio, 2022) have been provided. All datasets include both soft and hard labels as well as some additional information. Among these ArMIS, HS- Brexit, and MD-Agreement have a constant num- ber of annotators throughout the dataset, whereas ConvAbuse has a variable number of annotators. In this paper, we explore the effect of different activation functions in the output layer of a deep neural network model for the prediction of soft and hard labels. We introduce a sinusoidal activation function, which we refer to as the Sinusoidal Step Function (SSF). We evaluate the performance of the models using the SSF, sigmoid, as well as a post- training step function for soft-label, and therefore, hard-label prediction. The experiments indicate that the SSF activation achieves the highest F1-score for the hard-label prediction on the majority of the datasets. For the soft label prediction, the sigmoid activation obtains the …

Content truncated to 500 words