Get Semantic With Me! The Usefulness of Different Feature Types for Short-Answer Grading

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2186–2195, Osaka, Japan, December 11-17 2016. Get Semantic With Me! The Usefulness of Different Feature Types for Short-Answer Grading Ulrike Pad´o Hochschule f¨ur Technik Stuttgart Schellingstr. 24 70174 Stuttgart ulrike.pado@hft-stuttgart.de Abstract Automated short-answer grading is key to help close the automation loop for large-scale, comput- erised testing in education. A wide range of features on different levels of linguistic processing has been proposed so far. We investigate the relative importance of the different types of features across a range of standard corpora (both from a language skill and content assessment context, in English and in German). We find that features on the lexical, text similarity and dependency level often suffice to approximate full-model performance. Features derived from semantic processing particularly benefit the linguistically more varied answers in content assessment corpora. 1 Introduction Computerised testing is becoming ubiquitous in the educational domain, and automated and semi- automated grading of tests is in high demand to relieve the workload of teachers (especially in the context of Massive Open On-line Courses or repeated testing for continuous feedback during the academic year). NLP is a key technology to close or at last narrow the automation loop for grading of free-text answers and essays. We focus on the automated grading of short-answer questions (i.e., assessment questions that require a free-text answer up to two or three sentences in length). For this task, the training data consists of a question, at least one reference answer and several student answers. Systems then predict answer accuracy as a binary correct-incorrect decision or as a more fine-grained multi-class problem (or even a regression task predicting points). In contrast to the related essay grading task, correct student answers stay closer to the reference answer than good essays might to an example essay. As is often the case in young research areas, an important contribution was made by the Semeval-2013 shared task (Dzikovska et al., 2013), which introduced standard evaluation benchmarks. Two large data sets are now available with performance standards for system comparison. On the shared task data, researchers have experimented with various features based on linguistic processing, from syntactic information (used by a majority of entries in SemEval-2013, Dzikovska et al. (2013)) to deep semantic representations (Ott et al., 2013) or Textual Entailment (TE) systems (Zesch et al., 2013). Others, staying closer to the surface level, have recently experimented with sophisticated measures of textual similarity (Jimenez et al., 2013; Sultan et al., 2016) or inferring informative answer patterns (Ramachandran et al., 2015). However, similar to other NLP tasks (like, for example, TE), one of the biggest challenges remains beating the lexical baseline: At SemEval-2013, the baseline consisting of textual similarity measures comparing reference and student answer frequently was not outperformed. Given the large feature space proposed so far and the lack of consensus about where to find the most useful features, we ask whether there are any regularities in the predictiveness of features across corpora. …

Content truncated to 500 words