Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 752–762, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments Michael Mohler Dept. of Computer Science University of North Texas Denton, TX mgm0038@unt.edu Razvan Bunescu School of EECS Ohio University Athens, Ohio bunescu@ohio.edu Rada Mihalcea Dept. of Computer Science University of North Texas Denton, TX rada@cs.unt.edu Abstract In this work we address the task of computer- assisted assessment of short student answers. We combine several graph alignment features with lexical semantic similarity measures us- ing machine learning techniques and show that the student answers can be more accu- rately graded than if the semantic measures were used in isolation. We also present a first attempt to align the dependency graphs of the student and the instructor answers in order to make use of a structural component in the au- tomatic grading of student answers. 1 Introduction One of the most important aspects of the learning process is the assessment of the knowledge acquired by the learner. In a typical classroom assessment (e.g., an exam, assignment or quiz), an instructor or a grader provides students with feedback on their answers to questions related to the subject matter. However, in certain scenarios, such as a number of sites worldwide with limited teacher availability, on- line learning environments, and individual or group study sessions done outside of class, an instructor may not be readily available. In these instances, stu- dents still need some assessment of their knowledge of the subject, and so, we must turn to computer- assisted assessment (CAA). While some forms of CAA do not require sophis- ticated text understanding (e.g., multiple choice or true/false questions can be easily graded by a system if the correct solution is available), there are also stu- dent answers made up of free text that may require textual analysis. Research to date has concentrated on two subtasks of CAA: grading essay responses, which includes checking the style, grammaticality, and coherence of the essay (Higgins et al., 2004), and the assessment of short student answers (Lea- cock and Chodorow, 2003; Pulman and Sukkarieh, 2005; Mohler and Mihalcea, 2009), which is the fo- cus of this work. An automatic short answer grading system is one that automatically assigns a grade to an answer pro- vided by a student, usually by comparing it to one or more correct answers. Note that this is different from the related tasks of paraphrase detection and textual entailment, since a common requirement in student answer grading is to provide a grade on a certain scale rather than make a simple yes/no deci- sion. In this paper, we explore the possibility of im- proving upon existing bag-of-words (BOW) ap- proaches to short answer grading by utilizing ma- chine learning techniques. Furthermore, in an at- tempt to mirror the ability of humans to understand structural (e.g. syntactic) differences between sen- tences, we employ a rudimentary dependency-graph …

Content truncated to 500 words