ETS: Domain Adaptation and Stacking for Short Answer Scoring

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 275–279, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics ETS: Domain Adaptation and Stacking for Short Answer Scoring∗ Michael Heilman and Nitin Madnani Educational Testing Service 660 Rosedale Road Princeton, NJ 08541, USA {mheilman,nmadnani}@ets.org Abstract Automatic scoring of short text responses to educational assessment items is a challeng- ing task, particularly because large amounts of labeled data (i.e., human-scored responses) may or may not be available due to the va- riety of possible questions and topics. As such, it seems desirable to integrate various approaches, making use of model answers from experts (e.g., to give higher scores to responses that are similar), prescored student responses (e.g., to learn direct associations between particular phrases and scores), etc. Here, we describe a system that uses stack- ing (Wolpert, 1992) and domain adaptation (Daume III, 2007) to achieve this aim, allow- ing us to integrate item-specific n-gram fea- tures and more general text similarity mea- sures (Heilman and Madnani, 2012). We re- port encouraging results from the Joint Stu- dent Response Analysis and 8th Recognizing Textual Entailment Challenge. 1 Introduction In this paper, we address the problem of automati- cally scoring short text responses to educational as- sessment items for measuring content knowledge. Many approaches can be and have been taken to this problem—e.g., Leacock and Chodorow (2003), Nielsen et al. (2008), inter alia. The effectiveness of any particular approach likely depends on the the availability of data (among other factors). For exam- ple, if thousands of prescored responses are avail- ∗System description papers for SemEval 2013 are required to have a team ID (e.g., “ETS”) as a prefix. able, then a simple classifier using n-gram features may suffice. However, if only model answers (i.e., reference answers) or rubrics are available, more general semantic similarity measures (or even rule- based approaches) would be more effective. It seems likely that, in many cases, there will be model answers as well as a modest number of prescored responses available, as was the case for the Joint Student Response Analysis and 8th Rec- ognizing Textual Entailment Challenge (§2). There- fore, we desire to incorporate both task-specific fea- tures, such as n-grams, as well as more general fea- tures such as the semantic similarity of the response to model answers. We also observe that some features may them- selves require machine learning or tuning on data from the domain, in addition to any machine learn- ing required for the overall system. In this paper, we describe a machine learning ap- proach to short answer scoring that allows us to in- corporate both item-specific and general features by using the domain adaptation technique of Daume III (2007). In addition, the approach employs stacking (Wolpert, 1992) to support the integration of com- ponents that require tuning or machine learning. 2 Task Overview In this section, we describe the task to which we ap- plied our system: the …

Content truncated to 500 words