SemEval-2023 Task 4: ValueEval: Identification of Human Values Behind Arguments

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2287–2303 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 4: ValueEval: Identification of Human Values Behind Arguments Johannes Kiesel1,∗ Milad Alshomary2 Nailia Mirzakhmedova1 Maximilian Heinrich1 Nicolas Handke3 Henning Wachsmuth2 Benno Stein1 1 Bauhaus-Universität Weimar 2 Leibniz University Hannover 3 Leipzig University Abstract Argumentation is ubiquitous in natural lan- guage communication, from politics and media to everyday work and private life. Many ar- guments derive their persuasive power from human values, such as self-directed thought or tolerance, albeit often implicitly. These val- ues are key to understanding the semantics of arguments, as they are generally accepted as justifications for why a particular option is eth- ically desirable. Can automated systems un- cover the values on which an argument draws? To answer this question, 39 teams submitted runs to ValueEval’23. Using a multi-sourced dataset of over 9K arguments, the systems achieved F1-scores up to 0.87 (nature) and over 0.70 for three more of 20 universal value categories. However, many challenges remain, as evidenced by the low peak F1-score of 0.39 for stimulation, hedonism, face, and humility. 1 Introduction How come people disagree on the best course for- ward in controversial issues, even if they use the same information to form their opinion? We ob- serve that people have different beliefs and prior- ities of what is generally worth striving for (e.g., personal achievements vs. humility) and how to do so (e.g., being self-directed vs. respecting tradi- tions), often referred to as (human) values (Searle, 2003). Some values tend to conflict, others tend to align (see Figure 1). This can cause disagreement on the best course forward, but also the support, if not formation, of political parties that promote respective highly revered values. Due to their outlined importance, human val- ues have been studied both in the social sciences (Schwartz, 1994) and in formal argumentation (Bench-Capon, 2003) for decades. According to the former, a “value is a (1) belief (2) pertaining to desirable end states or modes of conduct, that ∗ Contact: johannes.kiesel@uni-weimar.de Level 1: 54 values Level 2: 20 value categories Humility Hedonism Face S el f- di re ct io n: C o nf or m it y: Stimulation thought action Achievement dominance resources personal societal Tradition rules interpersonal caring concern nature tolerance objectivity dependability B e n e v ol e n c e: U ni v er s al is m : S e c ur it y: P o w er : Figure 1: The employed taxonomy of 20 value cate- gories and their associated 54 values (shown as black dots), called Level 2 and Level 1 in Kiesel et al. (2022), respectively. ValueEval’23 focused exclusively on value categories. Categories that tend to conflict are placed on opposite sites. Taxonomy and illustration are largely adapted from Schwartz (1994) and subsequent works. (3) transcends specific situations, (4) guides selec- tion or evaluation of behavior, people, and events, and (5) is ordered by importance relative to …

Content truncated to 500 words