Sina at SemEval-2023 Task 4: A Class-Token Attention-based Model for Human Value Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 205–211 July 13-14, 2023 ©2023 Association for Computational Linguistics StFX-NLP at SemEval-2023 Task 4: Unsupervised and supervised approaches to detecting human values in arguments Ethan Heavey and Milton King and James Hughes eheavey@stfx.caSt. Francis Xavier University mking@stfx.ca jhughes@stfx.ca Abstract In this paper, we discuss our models applied to Task 4: Human Value Detection of SemEval 2023, which incorporated two different embed- ding techniques to interpret the data. Prelim- inary experiments were conducted to observe important word types. Subsequently, we ex- plored an XGBoost model, an unsupervised learning model, and two Ensemble learning models were then explored. The best perform- ing model, an ensemble model employing a soft voting technique, secured the 34th spot out of 39 teams, on a class imbalanced dataset. We explored the inclusion of different parts of the provided knowledge resource and found that considering only specific parts assisted our models. 1 Introduction SemEval-2023 Task 4: Human Value Detection fo- cuses on classifying a textual argument into one of 20 human value categories, each from Schwartz’ value continuum (Kiesel et al., 2023). Examples of categories include, Self-direction: thought (it is good to have one’s own ideas and interests), Face (it is good to maintain one’s public image), and Hu- mility (it is good to recognize one’s own insignif- icance in the larger scheme of things). Blending computer science and social science, the system can also be applied to a variety of topics outside of academia. Understanding the motives of oth- ers allows one to have more insightful conversa- tions as they approach topics from different angles (Crisp and Turner, 2011). By understanding what the other side of argument values, it can help us to reach common ground in discussions and lead to more productive interactions with less friction (Galinsky and Moskowitz, 2000). We approached this Task with models that con- sider both supervised and unsupervised techniques. Furthermore, we perform an analysis on the data provided but the task to observe trends among dif- ferent classes. The best performing model ranked 77th out of the 111 submitted. The code is open source and available on GitHub1. Our unsupervised learning model employed a cosine similarity comparison between record em- beddings and class description embeddings, taken from the knowledge resource2. To address some limitations of the system, the methods in which the data was preprocessed and embedded could have impacted the performance of our models. On top of data preprocessing, the dataset was only in English, limiting the reach of our models. 2 Background The submissions focused only on the ValueEval’23 dataset (Mirzakhmedova et al., 2023), which con- tains almost 9, 000 records, in English. Roughly 80% of the records came from IBM’s argument quality dataset (Gretz et al., 2020), 15% from the Conference for the Future of Europe (Barriere et al., 2022), and the remaining 5% from group discus- sion ideas. These arguments were then split by the Task organizers into separate training (60%), validation (20%), and …

Content truncated to 500 words