Your Answer is Incorrect... Would you like to know why? Introducing a Bilingual Short Answer Feedback Dataset

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8577 - 8591 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Your Answer is Incorrect... Would you like to know why? Introducing a Bilingual Short Answer Feedback Dataset Anna Filighera, Siddharth Singh Parihar, Sebastian Ochs, Tim Steuer and Tobias Meuser Multimedia Communications Lab Technical University of Darmstadt, Germany {anna.filighera, tim.steuer, tobias.meuser}@kom.tu-darmstadt.de {siddharth.masters, ochs.seb}@gmail.com Abstract Handing in a paper or exercise and merely re- ceiving "bad" or "incorrect" as feedback is not very helpful when the goal is to improve. Un- fortunately, this is currently the kind of feed- back given by many Automatic Short Answer Grading (ASAG) systems. One of the reasons for this is a lack of content-focused elaborated feedback datasets. To encourage research on explainable and understandable feedback sys- tems, we present the Short Answer Feedback dataset (SAF). Similar to other ASAG datasets, SAF contains learner responses and reference answers to German and English questions. However, instead of only assigning a label or score to the learners’ answers, SAF also con- tains elaborated feedback explaining the given score. Thus, SAF enables supervised train- ing of models that grade answers and explain where and why mistakes were made. This paper discusses the need for enhanced feed- back models in real-world pedagogical scenar- ios, describes the dataset annotation process, gives a comprehensive analysis of SAF, and provides T5-based baselines for future compar- ison.1 1 Introduction Assessment and feedback are essential to high- quality education (Shute, 2008). They allow learn- ers and teachers to discover misconceptions, gaps in knowledge, and improvement opportunities. However, manually assessing learners’ knowledge and providing helpful feedback is time-consuming and requires pedagogical as well as domain ex- pertise. Here, automatic assessment can free up teachers’ time to focus on tutoring learners or ade- quately preparing classroom activities. Moreover, it can be an alternative to peer-grading when course participant numbers increase beyond the financial 1Our code, scoring rubrics and dataset are available at https://github.com/SebOchs/SAF under an MIT license feasibility of manual grading (Kay et al., 2013), making it particularly useful for freely accessible online courses. Besides being cost- and time-efficient, automat- ing assessment also offers unique teaching opportu- nities. As long as systems give individual, response- specific feedback, learners may retry or take addi- tional assignments and receive instantaneous feed- back as often as they need. Additionally, knowing that a system instead of one’s teacher or professor will evaluate one’s assignment can also reduce anx- iety and help learners focus on their work instead of worrying about their reputation (Lipnevich and Smith, 2009). Therefore, it is unsurprising that automatic assessment has been an active research field over the past decades (Burrows et al., 2015; Ihantola et al., 2010; Ke and Ng, 2019; Xi, 2010). So far, significant progress has been made. In particular, Transformer models are approach- ing human experts’ performance on specific datasets in the Automatic Short Answer Grading (ASAG) field (Sung et al., 2019; Camus and Fil- ighera, 2020). …

Content truncated to 500 words