Effective Feature Integration for Automated Short Answer Scoring

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1049–1054, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Effective Feature Integration for Automated Short Answer Scoring∗ Keisuke Sakaguchi CLSP, Johns Hopkins University Baltimore, MD keisuke@cs.jhu.edu Michael Heilman Nitin Madnani Educational Testing Service Princeton, NJ {mheilman,nmadnani}@ets.org Abstract A major opportunity for NLP to have a real- world impact is in helping educators score stu- dent writing, particularly content-based writ- ing (i.e., the task of automated short answer scoring). A major challenge in this enterprise is that scored responses to a particular ques- tion (i.e., labeled data) are valuable for mod- eling but limited in quantity. Additional in- formation from the scoring guidelines for hu- mans, such as exemplars for each score level and descriptions of key concepts, can also be used. Here, we explore methods for in- tegrating scoring guidelines and labeled re- sponses, and we find that stacked generaliza- tion (Wolpert, 1992) improves performance, especially for small training sets. 1 Introduction Educational applications of NLP have considerable potential for real-world impact, particularly in help- ing to score responses to assessments, which could allow educators to focus more on instruction. We focus on the task of analyzing short, content- focused responses from an assessment of read- ing comprehension, following previous work on short answer scoring (Leacock and Chodorow, 2003; Mohler et al., 2011; Dzikovska et al., 2013). This task is typically defined as a text regression or clas- sification problem: we label student responses that consist of one or more sentences with scores on an ∗Work done when Keisuke Sakaguchi was an intern at ETS. Michael Heilman is now a data scientist at Civis Analytics. ordinal scale (e.g. correct, partially correct, or in- correct; 1–5 score range, etc.). Importantly, in ad- dition to the student response itself, we may also have available other information such as reference answers or descriptions of key concepts from the scoring guidelines for human scorers. Such informa- tion can be cheap to acquire since it is often gener- ated as part of the assessment development process. Generally speaking, most work on short answer scoring takes one of the following approaches: • A response-based approach uses detailed fea- tures extracted from the student response itself (e.g., word n-grams, etc.) and learns a scoring function from human-scored responses. • A reference-based approach compares the stu- dent response to reference texts, such as ex- emplars for each score level, or specifications of required content from the assessment’s scor- ing guidelines. Various text similarity methods (Agirre et al., 2013) can be used. These two approaches can, of course, be com- bined. However, to our knowledge, the issues of how to combine the approaches and when that is likely to be useful have not been thoroughly studied. A challenge in combining the approaches is that the response-based approach produces a large set of sparse features (e.g., word n-gram indicators), while the reference-based approach produces a small set of continuous …

Content truncated to 500 words