Automated Evaluation of Writing – 50 Years and Counting

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7796–7810 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7796 Automated Evaluation of Writing – 50 Years and Counting Beata Beigman Klebanov and Nitin Madnani Educational Testing Service, Princeton, NJ, USA {bbeigmanklebanov,nmadnani}@ets.org Abstract In this theme paper, we reflect on the progress of Automated Writing Evaluation (AWE), using Ellis Page’s seminal 1966 paper to frame the presenta- tion. We discuss some of the current frontiers in the field, and offer some thoughts on the emergent uses of this technology. 1 A Minimal Case for AWE In a seminal paper on the imminence of automated grading of essays, Page (1966) showed that a high correlation between holistic machine and human scores is possible. He demonstrated automated scoring of 276 essays written by high school stu- dents by a system with 32 features, resulting in a multiple R = 0.65 between machine and average human score, after adjustment. He also provided a thoughtful discussion of his ambitions for auto- mated scoring and of the possible objections. Page made the case that automated evaluation of student writing is needed to take some of the eval- uation load off the teachers and to provide students evaluations of their (potentially multiple) drafts with a fast turnaround. He then appealed to the then-burgeoning interest and fascination with ma- chine learning to argue for the feasibility of such an enterprise, namely, that machines can learn how to give the right grades to essays, if trained on an expert-scored sample. As part of the feasibility argument, Page em- phasized the need to carefully define the goal so that success can be judged appropriately. The goal is not a “real” master analysis of the essay the way a human reader would do but merely an imitation that would produce a correlated result (using what Page called proxes – approximations). Page con- sidered this goal to be both useful and achievable. 2 Report Card: Where are We Now? 2.1 Accomplishments Page’s minimal desiderata have certainly been achieved – AWE systems today can score in agree- ment with the average human rater, at least in some contexts.1 For example, Pearson’s Intelli- gent Essay Assessor™ (IEA) scores essays writ- ten for the Pearson Test of English (PTE) as well as for other contexts: “IEA was developed more than a decade ago and has been used to evaluate millions of essays, from scoring student writing at elementary, secondary and university level, to as- sessing military leadership skills.”2 Besides sole automated scoring as for PTE, there are additional contexts where the automated score is used in ad- dition to a human score, such as for essays written for the Graduate Record Examination (GRE®)3 or for the Test of English as a Foreign Language (TOEFL®).4 Does this mean that the problem of AWE is solved? Well, not exactly. 2.2 Needs Improvement Page did anticipate some difficulties for AWE sys- tems. It is instructive to see where we are with those. …

Content truncated to 500 words