Hitachi at SemEval-2023 Task 4: Exploring Various Task Formulations Reveals the Importance of Description Texts on Human Values

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1723–1735 July 13-14, 2023 ©2023 Association for Computational Linguistics Hitachi at SemEval-2023 Task 4: Exploring Various Task Formulations Reveals the Importance of Description Texts on Human Values Masaya Tsunokake, Atsuki Yamaguchi, Yuta Koreeda, Hiroaki Ozaki and Yasuhiro Sogawa Research and Development Group, Hitachi, Ltd. Kokubunji, Tokyo, Japan {masaya.tsunokake.qu, atsuki.yamaguchi.xn, yuta.koreeda.pb, hiroaki.ozaki.yu, yasuhiro.sogawa.tp}@hitachi.com Abstract This paper describes our participation in SemEval-2023 Task 4, ValueEval: Identifi- cation of Human Values behind Arguments. The aim of this task is to identify whether or not an input text supports each of the 20 pre-defined human values. Previous work on human value detection has shown the effec- tiveness of a sequence classification approach using BERT. However, little is known about what type of task formulation is suitable for the task. To this end, this paper explores various task formulations, including sequence classification, question answering, and ques- tion answering with chain-of-thought prompt- ing and evaluates their performances on the shared task dataset. Experiments show that a zero-shot approach is not as effective as other methods, and there is no one approach that is optimal in every scenario. Our analysis also re- veals that utilizing the descriptions of human values can help to improve performance. 1 Introduction SemEval-2023 Task 4 involves detecting human values behind argumentative sentences in English. Given a conclusion, stance, and premise, the task is to classify whether they entail each of the 20 pre-defined values (Kiesel et al., 2023). The perfor- mance improvement in this task can contribute to more precise analysis of argumentative texts using argument and opinion mining techniques. In this paper, we explore various task formula- tions for detecting human values in texts, including sequence classification, question answering, and question answering with chain-of-thought prompt- ing. Previous work on human value detection (Kiesel et al., 2022) thus far has only investigated the effectiveness of sequence classification using BERT (Devlin et al., 2019), and it is still unknown what type of task formulation is effective for human value detection. To this end, we test a variety of pretrained lan- guage models. For sequence classification, we em- ploy encoder-based models such as RoBERTa (Liu et al., 2019) and DeBERTaV3 (He et al., 2021). For question answering, we test both encoder-decoder and decoder models, including T5 (Raffel et al., 2020), BART (Lewis et al., 2020), and GPT-3 (text- davinci-003) (Brown et al., 2020). For question answering with chain-of-thought prompting, we use GPT-3 (text-davinci-003) in a zero-shot man- ner. We measure the detection performance of all approaches on three shared task datasets (Mirza- khmedova et al.), including the two optional test datasets: Nahj al-Balagha and New York Times. We also verify the effectiveness of three additional training strategies for sequence classification: loss weighting, pretraining on a similar corpus: Val- ueNet (Qiu et al., 2022), and adding a value de- scription to an input. In the competition1, our approach using RoBERTaLARGE trained with sequence classifica- tion and weighted loss achieved macro F1 scores of …

Content truncated to 500 words