Semi-Supervised Clustering for Short Answer Scoring

Semi-Supervised Clustering for Short Answer Scoring Andrea Horbach and Manfred Pinkal University of Duisburg-Essen & Saarland University, Germany andrea.horbach@uni-due.de, pinkal@coli.uni-saarland.de Abstract This paper investigates the use of semi-supervised clustering for Short Answer Scoring (SAS). In SAS, clustering techniques are an attractive alternative to classification because they provide structured groups of answers in addition to a score. Previous approaches use unsupervised clustering and have teachers label some items after clustering. We propose to re-allocate some of the human annotation effort to before and during the clustering process for (i) feature selection, (ii) for creating pairwise constraints and (iii) for metric learning. Our methods improve clustering performance substantially from 0.504 kappa for unsupervised clustering to 0.566. Keywords: Short-Answer Scoring, Clustering, Computer-Assisted Language Learning Prompt (ASAP dataset): During the story, the reader gets background information about Mr. Leonard. Explain the effect that background information has on Paul. Support your response with details from the story. Answer (2 points): Paul sees himself in Mr. Leonard. They both can’t read but both are good at track. Answer (0 points): He (the narrator) finds out that Mr. Leonard was once a star athelete at his college but dropped out because of grades. Prompt (PG dataset): What did the Declaration of Inde- pendence do? Answer (correct): declare independance from britain Answer (incorrect): protect the constitution and people Figure 1: Example prompts and answers 1. Introduction In an educational context, short answer questions are a type of exercise asking for short free-form answers of a few words or sentences in response to a given prompt. They oc- cur, for example, in the form of science questions or read- ing comprehension exercises and can address both native speakers, such as high school students, and foreign lan- guage learners. Automatic Short Answer Scoring (SAS) is the task of automatically assigning a score to such answers, either in numeric form (points) or by a binary true/false label. Figure 1. shows two example prompts with some scored answers. Automatic scoring of short answer exercises is a challenge: in contrast to multiple choice or simple gap-filling exer- cises, SAS has to assess the semantic correctness of an- swers, and it is thus related to fields such as natural lan- guage understanding, paraphrase detection or textual en- tailment. In addition, it has to deal with noisy user input containing spelling and grammar errors. Most approaches to SAS consider automatic scoring as a classification task, relying on supervised machine learning (ML) techniques which require manually labeledtraining data In contrast Basu et al. (2013),Brooks et al. (2014) and Horbach et al. (2014) have focused on the use of clustering techniques for SAS. The rationale behind this procedure is that answers that are similar to each other – and therefore end up in the same cluster – are also likely to receive the same label for scoring and can thus be scored in one grad- ing step by a teacher. Ideally, a teacher has to label only a single answer representative for the whole cluster which is …

Content truncated to 500 words