Text-to-Text Semantic Similarity for Automatic Short Answer Grading

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 567–575, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Text-to-text Semantic Similarity for Automatic Short Answer Grading Michael Mohler and Rada Mihalcea Department of Computer Science University of North Texas mgm0038@unt.edu, rada@cs.unt.edu Abstract In this paper, we explore unsupervised techniques for the task of automatic short answer grading. We compare a number of knowledge-based and corpus-based mea- sures of text similarity, evaluate the effect of domain and size on the corpus-based measures, and also introduce a novel tech- nique to improve the performance of the system by integrating automatic feedback from the student answers. Overall, our system significantly and consistently out- performs other unsupervised methods for short answer grading that have been pro- posed in the past. 1 Introduction One of the most important aspects of the learn- ing process is the assessment of the knowledge acquired by the learner. In a typical examination setting (e.g., an exam, assignment or quiz), this assessment implies an instructor or a grader who provides students with feedback on their answers to questions that are related to the subject mat- ter. There are, however, certain scenarios, such as the large number of worldwide sites with lim- ited teacher availability, or the individual or group study sessions done outside of class, in which an instructor is not available and yet students need an assessment of their knowledge of the subject. In these instances, we often have to turn to computer- assisted assessment. While some forms of computer-assisted assess- ment do not require sophisticated text understand- ing (e.g., multiple choice or true/false questions can be easily graded by a system if the correct so- lution is available), there are also student answers that consist of free text which require an analy- sis of the text in the answer. Research to date has concentrated on two main subtasks of computer- assisted assessment: the grading of essays, which is done mainly by checking the style, grammati- cality, and coherence of the essay (cf. (Higgins et al., 2004)), and the assessment of short student answers (e.g., (Leacock and Chodorow, 2003; Pul- man and Sukkarieh, 2005)), which is the focus of this paper. An automatic short answer grading system is one which automatically assigns a grade to an an- swer provided by a student through a comparison with one or more correct answers. It is important to note that this is different from the related task of paraphrase detection, since a requirement in stu- dent answer grading is to provide a grade on a cer- tain scale rather than a binary yes/no decision. In this paper, we explore and evaluate a set of unsupervised techniques for automatic short an- swer grading. Unlike previous work, which has either required the availability of manually crafted patterns (Sukkarieh et al., 2004; Mitchell et al., 2002), or large training data sets to bootstrap such patterns (Pulman and Sukkarieh, 2005), we at- tempt to devise an unsupervised …

Content truncated to 500 words