Fast and Easy Short Answer Grading with High Accuracy

Proceedings of NAACL-HLT 2016, pages 1070–1075, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Fast and Easy Short Answer Grading with High Accuracy Md Arafat Sultan Cristobal Salazar Institute of Cognitive ScienceTamara Sumner Department of Computer Science University of Colorado, Boulder, CO {arafat.sultan,crsa7687,sumner}@colorado.edu Abstract We present a fast, simple, and high-accuracy short answer grading system. Given a short- answer question and its correct answer, key measures of the correctness of a student re- sponse can be derived from its semantic sim- ilarity with the correct answer. Our super- vised model (1) utilizes recent advances in the identification of short-text similarity, and (2) augments text similarity features with key grading-specific constructs. We present exper- imental results where our model demonstrates top performance on multiple benchmarks. 1 Introduction Short-answer questions are a useful device for elic- iting student understanding of specific concepts in a subject domain. Numerous automated graders have been proposed for short answers based on their se- mantic similarity with one or more expert-provided correct answers (Mohler et al., 2011; Heilman and Madnani, 2013; Ramachandran et al., 2015). From an application perspective, these systems vary con- siderably along a set of key dimensions: amount of human effort involved, accuracy, speed, and ease of implementation. We explore a design that seeks to optimize performance along all these dimensions. Systems developed for the more general task of short-text semantic similarity provide a good start- ing point for such a design. Major progress has been made in this task in recent years, due primarily to the SemEval Semantic Textual Similarity (STS) task (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). However, the utility of top STS systems has remained largely unexplored in the context of short answer grading. We seek to bridge this gap by adopting the feature set of the best performing STS system at SemEval-2015 (Sultan et al., 2015). Besides high accuracy, this system also has a simple design and fast runtime. Textual similarity alone, however, is inadequate as a measure of answer correctness. For example, while the Sultan et al. (2015) system makes the gen- eral assumption that all content words1 contribute equally to the meaning of a sentence, domain key- words (e.g., “mutation” for biological evolution) are clearly more significant than arbitrary content words (e.g., “consideration”) for academic text. As another example, question demoting (Mohler et al., 2011) proposes discarding words that are present in the question text as a preprocessing step for grading. We augment our generic text similarity features with such grading-specific measures. We train supervised models with our final fea- ture set; in two different grading tasks, these mod- els demonstrate significant performance improve- ment over the state of the art. In summary, our contribution is a fast, simple, and high-performance short answer grading system which we also release as open-source software at: https://github. com/ma-sultan/short-answer-grader. 2 Related Work A comprehensive review of automatic short answer grading can be found in (Burrows et al., 2015). Here …

Content truncated to 500 words