John-Arthur at SemEval-2023 Task 4: Fine-Tuning Large Language Models for Arguments Classification

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1428–1432 July 13-14, 2023 ©2023 Association for Computational Linguistics John-Arthur at SemEval-2023 Task 4: Fine-Tuning Large Language Models for Arguments Classification Georgios Balikas Salesforce.com geompalik@hotmail.com Abstract This paper presents the system submissions of the John-Arthur team to the SemEval Task 4 “ValueEval: Identification of Human Values be- hind Arguments”. The best system of the team was ranked 3rd and the overall rank of the team was 2nd (the first team had the two best sys- tems). John-Arthur team models the ValueEval problem as a multi-class, multi-label text clas- sification problem. The solutions leverage re- cently proposed large language models that are fine-tuned on the provided datasets. To boost the achieved performance we employ differ- ent best practises whose impact on the model performance we evaluate here. The code is publicly available at github and the model on Huggingface hub. 1 Introduction SemEval Task 4 “ValueEval: Identification of Hu- man Values behind Arguments” has as a goal to classify a textual argument across one or more hu- man value categories (Kiesel et al., 2022, 2023). Understanding and identifying human values in ar- guments is a difficult task as they are implicit and their definitions are often vague. At the same time they are studied in various domains like social stud- ies and formal argumentation. As a result, having a system aiding to the task could have a measurable impact on such studies. To build an NLP system that classifies input text data among human values we build on recent work on large language models (LLMs). These models are capable of “understanding” English text and our hope is that they can be used to achieve satisfactory performance on the task. The ValueEval input data consist of three pieces of information: • The argument’s stance: which is “in favour” or “against” • The argument’s premise, and • The argument’s conclusion. On the other hand, the prediction step is to rec- ognize the human value for each input among 20 values. This is a multi-class, multi-label classifi- cation problem as most inputs belong to several human values. In our experiments we discovered that LLMs are a suitable solution to the task. Properly encoding the input information for the LLMs and fine-tuning the models with the data provided by the organiz- ers quickly improves the performance. Also, we found that bigger models achieve better results if tuned properly. Adding data to the training set also has a big impact on the obtained performance. We tried various ideas in the prototyping phase: from different ways to encode the model input to vari- ous architecture choices on how to use the model output. Compared to more traditional machine modeling lifecycle where a lot of effort is put on feature engineering, we often realized that while some feature engineering benefits the performance using smaller models, its impact on larger model is diminishing. The best system of the John-Arthur team is cur- rently ranked at position 3 …

Content truncated to 500 words