Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 431–439, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression Peter Phandi1 Kian Ming A. Chai2 Hwee Tou Ng1 1 Department of Computer Science, National University of Singapore {peter-p,nght}@comp.nus.edu.sg 2 DSO National Laboratories ckianmin@dso.org.sg Abstract Most of the current automated essay scor- ing (AES) systems are trained using manu- ally graded essays from a specific prompt. These systems experience a drop in accu- racy when used to grade an essay from a different prompt. Obtaining a large num- ber of manually graded essays each time a new prompt is introduced is costly and not viable. We propose domain adapta- tion as a solution to adapt an AES system from an initial prompt to a new prompt. We also propose a novel domain adapta- tion technique that uses Bayesian linear ridge regression. We evaluate our domain adaptation technique on the publicly avail- able Automated Student Assessment Prize (ASAP) dataset and show that our pro- posed technique is a competitive default domain adaptation algorithm for the AES task. 1 Introduction Essay writing is a common task evaluated in schools and universities. In this task, students are typically given a prompt or essay topic to write about. Essay writing is included in high-stakes as- sessments, such as Test of English as a Foreign Language (TOEFL) and Graduate Record Exami- nation (GRE). Manually grading all essays takes a lot of time and effort for the graders. This is what Automated Essay Scoring (AES) systems are try- ing to alleviate. Automated Essay Scoring uses computer soft- ware to automatically evaluate an essay written in an educational setting by giving it a score. Work related to essay scoring can be traced back to 1966 when Ellis Page created a computer grading software called Project Essay Grade (PEG). Re- search on AES has continued through the years. The recent Automated Student Assessment Prize (ASAP) Competition1 sponsored by the Hewlett Foundation in 2012 has renewed interest on this topic. The agreement between the scores assigned by state-of-the-art AES systems and the scores as- signed by human raters has been shown to be rel- atively high. See Shermis and Burstein (2013) for a recent overview of AES. AES is usually treated as a supervised machine learning problem, either as a classification, regres- sion, or rank preference task. Using this approach, a training set in the form of human graded essays is needed. However, human graded essays are not readily available. This is perhaps why research in this area was mostly done by commercial organi- zations. After the ASAP competition, research in- terest in this area has been rekindled because of the released dataset. Most of the recent AES related work is prompt- specific. That is, an AES system is trained using essays from a specific prompt and tested against essays from the same prompt. These AES systems will not work as well when tested …

Content truncated to 500 words