Mistake Captioning: A Machine Learning Approach for Detecting Mistakes and Generating Instructive Feedback

Proceedings of Recent Advances in Natural Language Processing, pages 1455–1462 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_163 1455 Mistake Captioning: A Machine Learning Approach For Detecting Mistakes and Generating Instructive Feedback Anton Vinogradov University of Kentucky Lexington, KY 40506 anton.vinogradov@uky.edu Andrew Miles Byrd University of Kentucky Lexington, KY 40506 andrewbyrd@uky.edu Brent Harrison University of Kentucky Lexington, KY 40506 harrison@cs.uky.edu Abstract Giving feedback to students is not just about marking their answers as correct or incorrect, but also finding mistakes in their thought pro- cess that led them to that incorrect answer. In this paper, we introduce a machine learning technique for mistake captioning, a task that attempts to identify mistakes and provide feed- back meant to help learners correct these mis- takes. We do this by training a sequence-to- sequence network to generate this feedback based on domain experts. To evaluate this system, we explore how it can be used on a Linguistics assignment studying Grimm’s Law. We show that our approach generates feedback that outperforms a baseline on a set of auto- mated NLP metrics. In addition, we perform a series of case studies in which we examine successful and unsuccessful system outputs. 1 Introduction Giving feedback is one of the most critical parts of education and training. It allows the instructor to correct errors in students’ understanding and guide them to the correct solution. In automated learning systems, such as intelligent tutoring systems (ITSs) the feedback and hints are focused on getting the student to complete the assignment, but this may not take into account their mastery of that skill. While better feedback can be made by considering student intent, we find that current models are lack- ing in this regard. They are typically focused on learning objective mastery and may not necessar- ily give the whole picture when determining why students answered something the way they did. But there is a natural solution to this, in that skilled educators are capable of intuitively mod- eling a student’s thought process and determining mistakes in it. This skill is often developed over time. The difficulty of this task for educators de- pends on the field of study, with some more free form fields like writing and programming being particularly difficult. By indirectly modeling a stu- dent’s intent by instead modeling how the instructor gives feedback, we can use this model to generate novel feedback sentences. We propose a system that when given a mis- take that has been identified in the task of deriving sound changes from Proto-Indo-European to Proto- Germanic, will automatically generate an explana- tion that identifies the mistake and the reasoning to why this sound change should not apply in this case. We call this method Mistake Captioning as it takes inspiration from image captioning meth- ods and post-hoc AI rationale generation methods such as Ehsan et al. (2019) Automated Rationale Generation. We apply a suite of automated machine transla- tion and image captioning metrics to test our sys- tem. We also take a closer look at 3 cases …

Content truncated to 500 words