ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores

ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores Sandeep Mathias, Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology, Bombay {sam, pb}@cse.iitb.ac.in Abstract In this paper, we describe the creation of a resource - ASAP++ - which is basically annotations of the Automatic Student Assessment Prize’s Automatic Essay Grading dataset. These annotations are scores for different attributes of the essays, such as content, word choice, organization, sentence fluency, etc. Each of these essays is scored by an annotator. We also report the results of each of the attributes using a Random Forest Classifier using a baseline set of attribute independent features as described by Zesch et al. (2015). We release and share this resource to facilitate further research into these attributes of essay grading. Keywords: Automatic Essay Grading, Attribute-specific Essay Grading 1. Introduction Automatic essay grading (AEG) is one of the most chal- lenging activities in natural language processing (NLP). AEG makes use of many NLP and machine learning (ML) techniques in predicting the score of an essay - a piece of text that is written by a human on a given topic (called a prompt). It has been around since the 1960s, with the first AEG system - Project Essay Grade - proposed by Ellis Page (Page, 1966). Since then, there have been multiple systems that look at providing either a holistic score to the essay, or to score individual attributes of the essay. Examples of a few online systems include Grammarly1 and Paper-Rater2. Essay grading systems rely on training and test data in or- der to grade essays. Most of the research today makes use of the Automated Student Assessment Prize’s (ASAP) Automated Essay Grading (AEG) dataset to train and test systems3. The ASAP AEG dataset comprises of approxi- mately 13,000 essays, written across 8 prompts. The essays were written by students of class 7 to 10. Each essay was evaluated by 2 evaluators. 6 out of the 8 prompts only have overall scores. Only 2 of them have scores for individual essay attributes, like content, organization, style, etc. Our contribution is the scoring of individual attributes of the essays, like content, organization, style, etc. in the ASAP dataset for the remainder of the essays. 2. Motivation A lot of the work in essay grading today makes use of the ASAP AEG dataset. However, most of the essays only have an overall score, not attribute-specific scores. This limita- tion limits the utility of this dataset for predicting the scores of particular attributes of essays. Shermis and Burstein (2013) in chapter 19 (Contrasting State-of-the-Art Automatic Essay Grading Systems) of their book describe the ASAP dataset, as well as the results of current commercial AEG systems in scoring those es- says. Since then, a large amount of work has been done 1www.grammarly.com 2www.paperrater.com 3The dataset can be downloaded from here: https://www. kaggle.com/c/asap-aes/data. using that dataset for evaluating the overall score of es- says, from using machine learning techniques (Chen and He, 2013; Phandi …

Content truncated to 500 words