Arguably at SemEval-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2137–2142 July 13-14, 2023 ©2023 Association for Computational Linguistics Arguably at SemEval-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models Guneet Kohli Thapar University guneetsk99@gmail.com Vinayak Tiwari Netaji Subhas University of Technology vinayaktiwari2897@gmail.com Abstract We describe SemEval-2023 Task 11, on behav- ioral segregation of annotations to find the sim- ilarities and contextual thinking of a group of annotators. We have utilized a behavioral seg- mentation analysis on the annotators to model them independently and combine the results to yield soft and hard scores. Our team focused on experimenting with hierarchical clustering with various distance metrics for similarity, dissimi- larity, and reliability. We modeled the clusters and assigned weightage to find the soft and hard scores. Our team was able to find out hid- den behavioral patterns among the judgments of annotators after rigorous experiments. The proposed system is made available 1 1 Introduction In the field of natural language processing (NLP), it has been assumed that a given natural language expression (such as a sentence) has a single, clear and unambigous meaning in a particular context. However, this assumption is now being recognized as an idealization or simplification that does not al- ways hold in practice. For example, the sentence "I saw her duck" could mean that someone observed a woman’s pet duck, or that someone watched a woman quickly lower her head to avoid hitting it on a low object. In these cases, the interpretation of the sentence depends on the context and the particular meanings of the words used. The Learning with Disagreement shared task (Leonardellli et al., 2023) aims to provide a test- ing framework for machine learning models that can learn from disagreements in natural language interpretation. Our approach involves developing a model that can take into account multiple possi- ble interpretations of a single sentence and identify the most likely one based of available context and information. 1https://github.com/guneetsk99/LeWiDi_Arguably We have utilized unsupervised learning approach and experimented using various clustering tech- niques and different types of similarities of anno- tations and metadata to bring up associations and relationships among various groups of annotators. This could tell us how a specific group of annota- tors think given a particular scenario. 2 Background This shared task provides a testing framework for learning from disagreements using datasets contain- ing information about disagreements for interpret- ing language. The focus of this task is on subjective tasks, where training with aggregated labels is less effective. There are four textual datasets with dif- ferent characteristics, including social media and conversation genres, English and Arabic languages, and tasks related to misogyny, hate speech, and of- fensiveness detection. These datasets also use dif- ferent annotation methodologies, including experts, specific demographic groups, and AMT-crowd, and all provide multiple labels for each instance. The Le-Wi-Di dataset is a collection of four ex- isting datasets that have been harmonized into a common json format that emphasizes their com- monalities. These datasets …

Content truncated to 500 words