PAI at SemEval-2023 Task 4: A General Multi-label Classification System with Class-balanced Loss Function and Ensemble Module

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 256–261 July 13-14, 2023 ©2023 Association for Computational Linguistics PAI at SemEval-2023 Task 4: A general multi-label classification system with class-balanced loss function and ensemble module Long Ma, Zeye Sun, Jiawei Jiang, Xuan Li {malong633,sunzeye273,jiangjiawei358,lixuan208}@pingan.com.cn Ping An Life Insurance Company of China, Ltd. Abstract The Human Value Detection shared task (Kiesel et al., 2023) aims to classify whether or not the argument draws on a set of 20 value categories, given a textual argument. This is a difficult task as the discrimination of human val- ues behind arguments is often implicit. More- over, the number of label categories can be up to 20 and the distribution of data is highly imbalanced. To address these issues, we em- ploy a multi-label classification model and uti- lize a class-balanced loss function. Our system wins 5 first places, 2 second places, and 6 third places out of 20 categories of the Human Value Detection shared task, and our overall average score of 0.54 also places third. The code is pub- licly available at https://www.github.com/ diqiuzhuanzhuan/semeval2023. 1 Introduction The Human Value Detection shared task (Kiesel et al., 2023) aims at building a system to identify human values behind a textual argument. Though human values have been considered in formal argu- mentation for about 20 years (Bench-Capon, 2003), it is the first task to draw values from arguments computationally. The organizers provide the first dataset Touché23-ValueEval Dataset (Mirzakhme- dova et al., 2023) for studying human values behind arguments. It contains 5270 arguments extracted from four different cultural backgrounds: Africa, China, India, and USA. All arguments are written in English. Every argument consists of three parts: a premise, a conclusion, and a stance indicating whether the premise is in favor of or against the conclusion. Table 1 demonstrates examples from the dataset. In this paper, we propose a general multi-label classification system for the Human Value De- tection shared task. To handle imbalanced data, we apply a class-balanced loss function and per- form data augmentation for classes with fewer data. Transformer-based models have demonstrated re- markable performance across a variety of NLP tasks in recent years. As a result, we adopt these models as encoder in our work. To select the best encoder model, we evaluate 4 candidates, includ- ing BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), De- BERTa (He et al., 2020) and etc. The sentence vector computed by the encoder is used as input for the multi-label classifier. We select Sigmoid function as the activation function for the classi- fier. Given the highly imbalanced nature of the data, we experiment with 5 loss functions that are specifically designed for such case, including focal loss, distribution-balanced loss and etc., and se- lect CB-NTR loss (Huang et al., 2021) as our final loss function. In terms of data preprocessing, we conduct rewriting and data augmentation to solve the weaknesses in the data and the model. In …

Content truncated to 500 words