Argument Similarity Assessment in German for Intelligent Tutoring: Crowdsourced Dataset and First Experiments

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2177–2187 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2177 Argument Similarity Assessment in German for Intelligent Tutoring: Crowdsourced Dataset and First Experiments Xiaoyu Bai, Manfred Stede Applied Computational Linguistics, University of Potsdam Karl-Liebknecht-Straße 24-25, 14476, Potsdam, Germany {xiaoyu.bai, stede}@uni-potsdam.de Abstract NLP technologies such as text similarity assessment, question answering and text classification are increasingly being used to develop intelligent educational applications. The long-term goal of our work is an intelligent tutoring system for German secondary schools, which will support students in a school exercise that requires them to identify arguments in an argumentative source text. The present paper presents our work on a central subtask, viz. the automatic assessment of similarity between a pair of argumentative text snippets in German. In the designated use case, students write out key arguments from a given source text; the tutoring system then evaluates them against a target reference, assessing the similarity level between student work and the reference. We collect a dataset for our similarity assessment task through crowdsourcing as authentic German student data are scarce; we label the collected text pairs with similarity scores on a 5-point scale and run first experiments on the task. We see that a model based on BERT shows promising results, while we also discuss some challenges that we observe. Keywords: Semantic textual similarity, Argument similarity, Intelligent tutoring systems 1. Introduction Recent years have seen increasing interest in apply- ing natural language processing (NLP) applications to the field of education. Topics such as automated es- say scoring (Uto et al., 2020), grammatical error cor- rection (Bryant et al., 2019) and automated writing as- sistance (Zhang et al., 2019; Madnani et al., 2018) have attracted much attention. While focus has been on English, research has also be done on other lan- guages, including Chinese (Gong et al., 2021), Spanish (Gonz´alez-L´opez et al., 2020), Japanese (Mizumoto et al., 2019) and German (Horbach et al., 2017). In the context of intelligent tutoring in German, we aim to implement an NLP-based system that supports secondary school students in a common source-based argumentative writing exercise known in German as textgebundene Er¨orterung1 (TE). In TE, students read a text discussing a controversial social topic, such as whether or not German universities should introduce tuition fees. Students then compose an essay in which they analyse the arguments given by the author of the text and illustrate their own position on the topic. A preparation for writing the essay is the correct identifi- cation of the arguments found in the source text, guided by questions such as what is the author’s core mes- sage? or what arguments are presented by the author to support their stance? Writing out the main ideas and arguments in the source text is commonly considered as a useful first-step exercise before producing the actual essay.2 Our long-term goal is to implement a tutoring system 1Roughly meaning ”text-bound argumentation”. 2See e.g. exam …

Content truncated to 500 words