TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2107–2111 July 13-14, 2023 ©2023 Association for Computational Linguistics TeamEC at SemEval-2023 Task 4: Transformers VS. Low-Resource Dictionaries, Expert Dictionary VS. Learned Dictionary Nicolas Stefanovitch Joint Research Centre European Commission nicolas.stefanovitch@ec.europa.eu Mario Scharfbillig Joint Research Centre European Commission mario.scharfbillig@ec.europa.eu Bertrand de Longueville Joint Research Centre European Commission bertrand.de-longueville@ec.europa.eu Abstract This paper describes the system we used to par- ticipate in the shared task (Kiesel et al., 2023), as well as additional experiments beyond the scope of the shared task, but using its data. Our primary goal is to compare the effectiveness of transformers model compared to low-resource dictionaries. Secondly, we compare the dif- ference in performance of a learned dictionary and of a dictionary designed by experts in the field of values. Our findings surprisingly show that transformers perform on par with a dic- tionary containing less than 1k words, when evaluated with 19 fine-grained categories, and only outperform a dictionary-based approach in a coarse setting with 10 categories. Inter- estingly, the expert dictionary has a precision on par with the learned one, while its recall is clearly lower, potentially an indication of over- fitting of topics to values in the shared task’s dataset. Our findings should be of interest to both the NLP and Value scientific communities on the use of automated approaches for value classification. 1 Introduction and Background This paper describes our participating system to the ValueEval shared task (SemEval 2023 task 4), which can be described as follows : “Given a tex- tual argument and a human value category, classify whether or not the argument draws on that cate- gory.” (Kiesel et al., 2023). The dataset for this task was built from existing datasets with the aim to cover a wide range of cultural sensitives, while English is the only language considered (Mirza- khmedova et al., 2023). Automated detection of Human Values is seen by the authors as an important tool in a broader policy analysis toolbox. Indeed, recent research demonstrated that the design and acceptance of public policies is influenced by their underlying values and the way they are framed in values terms (Mair et al., 2019; Scharfbillig et al., 2021). Our aim in participating in this task was to further explore the possible interplay between dictionary-based techniques - which have policy- related application (e.g. Scharfbillig et al., 2022)- learned dictionary approaches based on keyword extraction (Beliga, 2014) and more state-of-the- art methods involving large language models (Liu et al., 2019). Our contribution consists of: • Using the task’s annotated training set, ex- tracting representative keywords for each of the 19 values classes, and using such learned dictionary to make a classifier using simple keyword-matching heuristics; • Compare the results with fine-tuned RoBERTa transformers using the same data; • Compare an expert designed dictionary and a learned one over a coarse grained setting, using only 10 values classes; By doing so, we aim to stimulate debate and un- derstanding on the respective merits …

Content truncated to 500 words