Powergrading: a Clustering Approach to Amplify Human Effort for Short Answer Grading

Powergrading: a Clustering Approach to Amplify Human Effort for Short Answer Grading Sumit Basu Chuck Jacobs Lucy Vanderwende Microsoft Research Microsoft Research Microsoft Research One Microsoft Way One Microsoft Way One Microsoft Way Redmond, WA Redmond, WA Redmond, WA sumitb@microsoft.com cjacobs@microsoft.com lucyv@microsoft.com Abstract We introduce a new approach to the ma- chine-assisted grading of short answer ques- tions. We follow past work in automated grading by first training a similarity metric between student responses, but then go on to use this metric to group responses into clus- ters and subclusters. The resulting group- ings allow teachers to grade multiple re- sponses with a single action, provide rich feedback to groups of similar answers, and discover modalities of misunderstanding among students; we refer to this amplifica- tion of grader effort as “powergrading.” We develop the means to further reduce teacher effort by automatically performing actions when an answer key is available. We show results in terms of grading progress with a small “budget” of human actions, both from our method and an LDA-based approach, on a test corpus of 10 questions answered by 698 respondents. 1 Introduction Increasing access to quality education is a global is- sue, and one of the most exciting developments in recent years has been the introduction of MOOCs— massively online open courses, in which hundreds or thousands of students take a course online. While this works wonderfully for lectures, assessment in the form of quizzes and exams presents some sig- nificant challenges. One straightforward solution is to use multiple choice questions, but it is well known that there is far greater educational benefit from fill-in-the-blank and essay questions (Ander- son and Biddle, 1975). For many domains, then, there could be great value in using short answer questions; the problem is grading those answers without making the cost prohibitive. Even in rela- tively small classrooms of a few dozen to a few hun- dred, the ability to grade such answers more effi- ciently would be a great boon to teachers. One approach to addressing this is to automati- cally grade such answers as right or wrong or with a numerical score—there have been significant past efforts in this space (Leacock and Chodorow, 2003; Mohler and Mihalcea, 2009; Jordan and Mitchell, 2009). However, in practice this path has some sig- nificant obstacles. The first is that while these ap- proaches have made impressive progress, they are never 100% accurate—some number of misgraded answers are left on the table. The second is that as- signing a score is not really sufficient—in a small classroom, a teacher would give feedback as to why the answer is wrong; ideally she would be able to do this in the MOOC scenario as well. A third problem, or at least a lost opportunity, is that there may be consistent patterns of misunderstanding amongst students that go beyond the fraction getting a ques- tion right or wrong. For instance many students might mistakenly believe that one of the rights af- forded by the first …

Content truncated to 500 words