Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 532–541 July 13-14, 2023 ©2023 Association for Computational Linguistics Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models Daniel Schroter, Daryna Dementieva and Georg Groh TU Munich, Department of Informatics, Germany daniel@schroter.biz, daryna.dementieva@tum.de, grohg@in.tum.de Abstract This paper presents the best-performing ap- proach alias "Adam Smith" for the SemEval- 2023 Task 4: "Identification of Human Values behind Arguments". The goal of the task was to create systems that automatically identify the values within textual arguments. We train transformer-based models until they reach their loss minimum or f1-score maximum. Ensem- bling the models by selecting one global de- cision threshold that maximizes the f1-score leads to the best-performing system in the com- petition. Ensembling based on stacking with logistic regressions shows the best performance on an additional dataset provided to evaluate the robustness ("Nahj al-Balagha"). Apart from outlining the submitted system, we demonstrate that the use of the large ensemble model is not necessary and that the system size can be sig- nificantly reduced. 1 Introduction "We should ban whaling, whaling is wiping out species for little in return." say some, "We shouldn’t, it is part of a great number of cultures." say others. Both arguments support their claim, but why are some arguments more convincing to us than others? This might relate to the underlying values they address. Whereas the first argument ap- peals to the value of "universalism: nature", the sec- ond one addresses the value of "tradition". Whether an argument is in agreement or disagreement with our values influences its ability to persuade. The task organizers (Kiesel et al., 2023) are the first who extend the field of argument mining by this "value" dimension. As part of the SemEval-2023 workshop, they organize the task of automatically detecting human values behind arguments. They decided to add two additional test datasets to evalu- ate the robustness of the developed systems (Mirza- khmedova et al., 2023). Our system uses an ensemble of transformer- based models that are either trained until they reach their loss minimum or the f1-score maximum. To ensemble the models we average the individual predictions and calculate a decision threshold for the final system on a separate "Leave-out-Dataset". This model achieves the best performance in the competition ("Main" dataset). Each team was al- lowed to submit up to four systems. Ensembling the predictions by using stacking with logistic re- gressions leads to the system with the best perfor- mance on the additional "Nahj al-Balagha" dataset. In this paper, we describe the best-performing sys- tem on the "Main" dataset and briefly outline the ideas behind the other submitted systems. The system can be accessed through a web-based interface that is available online1. Furthermore, a docker container, models, and code are open source and publicly accessible (Appendix A). 2 Background Mirzakhmedova et al. (2023) created a labeled dataset of 9324 arguments from 6 different sources. The arguments are structured as follows: Premise whaling …

Content truncated to 500 words