ESCRITO - An NLP-Enhanced Educational Scoring Toolkit

ESCRITO – An NLP-Enhanced Educational Scoring Toolkit Torsten Zesch and Andrea Horbach Language Technology Lab, Department of Computer Science and Applied Cognitive Science University of Duisburg-Essen, Germany {torsten.zesch, andrea.horbach}@uni-due.de Abstract We propose ESCRITO, a toolkit for scoring student writings using NLP techniques that addresses two main user groups: teachers and NLP researchers. Teachers can use a high-level API in the teacher mode to assemble scoring pipelines easily. NLP researchers can use the developer mode to access a low-level API, which not only makes available a number of pre-implemented components, but also allows the user to integrate their own readers, preprocessing components, or feature extractors. In this way, the toolkit provides a ready-made testbed for applying the latest developments from NLP areas like text similarity, paraphrase detection, textual entailment, and argument mining within the highly challenging task of educational scoring and feedback. At the same time, it allows teachers to apply cutting-edge technology in the classroom. Keywords: automatic assessment, text classification und software toolkit 1. Introduction Scoring student writings is a core task for teachers, which requires a lot of manual effort. Using assisted or auto- mated scoring might have a tremendous impact on the qual- ity of teaching, as it potentially shifts the focus from te- dious assessment tasks to communicating knowledge. In the case of free-form student writings, like essays or an- swers to factual questions, correctly assessing a response is still a challenging task. There are typically many ways how a correct response can be expressed, and the variety of responses is increased even more by orthographic and grammatical deviations, which occur frequently in student writings. Therefore, automatically scoring student writings usually involves various other tasks central to NLP such as spell checking, grammatical error correction, POS tag- ging, paraphrase recognition, textual entailment, or argu- ment mining. In recent years, attempts have been made to link free- form scoring to some of the above-mentioned fields. For example, the SemEval 2013 Student Response Analysis Task (Dzikovska et al., 2013) combined short-answer scor- ing with recognizing textual entailment. More fundamental research along such lines would be desirable. In addition to basic research, more practical experimentation in the class- room is needed as well. These goals require the involve- ment of two disjunct groups of people: NLP researchers (who develop scoring methods) and teachers (who bring the methods to the classroom). However, existing imple- mentations are often very specific to certain use cases and datasets. That makes it difficult for teachers without a tech- nical background to use them out-of-the-box, let alone ap- ply them to new data. At the same time, the application field is highly complex, which discourages NLP researchers from testing the latest developments in this field. We thus present ESCRITO, the Educational SCoRIng TOolkit, a toolbox for free-text scoring based on natural language processing and machine learning, which caters to both user groups. ESCRITO has two main goals: (i) to en- able teachers to quickly build free-text scoring systems and apply them in real-life scenarios, and (ii) …

Content truncated to 500 words