A Corpus for Suggestion Mining of German Peer Feedback

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5539–5547 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5539 A Corpus for Suggestion Mining of German Peer Feedback Dominik Pf¨utze, Eva Ritz, Julius Janda, Roman Rietsche Institute of Information Management, University of St.Gallen M¨uller-Friedberg-Str. 8, 9000 St.Gallen, Switzerland {dominik.pfuetze, eva.ritz, julius.janda, roman.rietsche}@unisg.ch Abstract Peer feedback in online education becomes increasingly important to meet the demand for feedback in large scale classes, such as e.g. Massive Open Online Courses (MOOCs). However, students are often not experts in how to write helpful feedback to their fellow students. In this paper, we introduce a corpus compiled from university students’ peer feedback to be able to detect suggestions on how to improve the students’ work and therefore being able to capture peer feedback helpfulness. To the best of our knowledge, this corpus is the first student peer feedback corpus in German which additionally was labelled with a new annotation scheme. The corpus consists of more than 600 written feedback (about 7,500 sentences). The utilisation of the corpus is broadly ranged from Dependency Parsing to Sentiment Analysis to Suggestion Mining, etc. We applied the latter to empirically validate the utility of the new corpus. Suggestion Mining is the extraction of sentences that contain suggestions from unstructured text. In this paper, we present a new annotation scheme to label sentences for Sug- gestion Mining. Two independent annotators labelled the corpus and achieved an inter-annotator agreement of 0.71. With the help of an expert arbitrator a gold standard was created. An automatic classification using BERT achieved an accuracy of 75.3%. Keywords: Peer Feedback, Suggestion Mining, Corpus, Annotation 1. Introduction In the educational domain, large scale lectures often lack the possibility of providing personalised feed- back, which is caused by scarce resources (Rietsche & S¨ollner, 2019). Further, the recent rise of technology- mediated learning environments, i.e. MOOCs impedes the student-per-educator ratio and educators struggle to give individual feedback to their students. Recently, one phenomenon used to overcome this issue are peer- review processes between students (Pastor & Baruf- faldi, 2020). However, students are usually non experts in how to structure feedback, that it becomes most help- ful for the feedback receivers to improve their work. Therefore, we introduced a feedback process for stu- dents within a master’s university course to develop our corpus to detect suggestions on how to improve ones work. In a peer feedback setting, students give each other feedback on lecture-based tasks in text form. By using the corpus every semester, we can continuously expand the source texts and feedback. More detailed, we aim to extract information from the feedback and apply suggestion mining based on the corpus. In general, the relevance of suggestion mining from unstructured text increased due to the large availabil- ity of reviews and feedback in different forms online (Viswanathan et al., 2011). Recent work has been con- ducted on mining texts to extract user sentiments or opinions and analyse sentiments to …

Content truncated to 500 words