A New Dataset and Method for Automatically Grading ESOL Texts

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 180–189, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics A New Dataset and Method for Automatically Grading ESOL Texts Helen Yannakoudakis Computer Laboratory University of Cambridge United Kingdom Helen.Yannakoudakis@cl.cam.ac.uk Ted Briscoe Computer Laboratory University of Cambridge United Kingdom Ted.Briscoe@cl.cam.ac.uk Ben Medlock iLexIR Ltd Cambridge United Kingdom ben@ilexir.co.uk Abstract We demonstrate how supervised discrimina- tive machine learning techniques can be used to automate the assessment of ‘English as a Second or Other Language’ (ESOL) examina- tion scripts. In particular, we use rank prefer- ence learning to explicitly model the grade re- lationships between scripts. A number of dif- ferent features are extracted and ablation tests are used to investigate their contribution to overall performance. A comparison between regression and rank preference models further supports our method. Experimental results on the first publically available dataset show that our system can achieve levels of performance close to the upper bound for the task, as de- fined by the agreement between human exam- iners on the same corpus. Finally, using a set of ‘outlier’ texts, we test the validity of our model and identify cases where the model’s scores diverge from that of a human examiner. 1 Introduction The task of automated assessment of free text fo- cuses on automatically analysing and assessing the quality of writing competence. Automated assess- ment systems exploit textual features in order to measure the overall quality and assign a score to a text. The earliest systems used superficial features, such as word and sentence length, as proxies for understanding the text. More recent systems have used more sophisticated automated text processing techniques to measure grammaticality, textual co- herence, prespecified errors, and so forth. Deployment of automated assessment systems gives a number of advantages, such as the reduced workload in marking texts, especially when applied to large-scale assessments. Additionally, automated systems guarantee the application of the same mark- ing criteria, thus reducing inconsistency, which may arise when more than one human examiner is em- ployed. Often, implementations include feedback with respect to the writers’ writing abilities, thus fa- cilitating self-assessment and self-tutoring. Implicitly or explicitly, previous work has mostly treated automated assessment as a supervised text classification task, where training texts are labelled with a grade and unlabelled test texts are fitted to the same grade point scale via a regression step applied to the classifier output (see Section 6 for more de- tails). Different techniques have been used, includ- ing cosine similarity of vectors representing text in various ways (Attali and Burstein, 2006), often com- bined with dimensionality reduction techniques such as Latent Semantic Analysis (LSA) (Landauer et al., 2003), generative machine learning models (Rudner and Liang, 2002), domain-specific feature extraction (Attali and Burstein, 2006), and/or modified syntac- tic parsers (Lonsdale and Strong-Krause, 2003). A recent review identifies twelve different auto- mated free-text scoring systems (Williamson, 2009). Examples include e-Rater (Attali and Burstein, 2006), Intelligent Essay Assessor (IEA) (Landauer et al., 2003), IntelliMetric …

Content truncated to 500 words