Constrained Multi-Task Learning for Automated Essay Scoring

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 789–799, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Constrained Multi-Task Learning for Automated Essay Scoring Ronan Cummins ALTA Institute Computer Lab University of Cambridge Meng Zhang ALTA Institute Computer Lab University of Cambridge {rc635,mz342,ejb}@cl.cam.ac.uk Ted Briscoe ALTA Institute Computer Lab University of Cambridge Abstract Supervised machine learning models for automated essay scoring (AES) usually re- quire substantial task-specific training data in order to make accurate predictions for a particular writing task. This limita- tion hinders their utility, and consequently their deployment in real-world settings. In this paper, we overcome this shortcoming using a constrained multi-task pairwise- preference learning approach that enables the data from multiple tasks to be com- bined effectively. Furthermore, contrary to some recent re- search, we show that high performance AES systems can be built with little or no task-specific training data. We perform a detailed study of our approach on a pub- licly available dataset in scenarios where we have varying amounts of task-specific training data and in scenarios where the number of tasks increases. 1 Introduction Automated essay scoring (AES) involves the pre- diction of a score (or scores) relating to the quality of an extended piece of written text (Page, 1966). With the burden involved in manually grading stu- dent texts and the increase in the number of ESL (English as a second language) learners world- wide, research into AES is increasingly seen as playing a viable role in assessment. Automating the assessment process is not only useful for ed- ucators but also for learners, as it can provide in- stant feedback and encourage iterative refinement of their writing. The AES task has usually been addressed using machine learning. Given a set of texts and associ- ated gold scores, machine learning approaches aim to build models that can generalise to unseen in- stances. Regression (Page, 1994; Persing and Ng, 2014; Phandi et al., 2015), classification (Larkey, 1998; Rudner and Liang, 2002), and preference- ranking1 approaches (Yannakoudakis et al., 2011) have all been applied to the task. In general, ma- chine learning models only perform well when the training and test instances are from similar dis- tributions. However, it is usually the case that essays are written in response to prompts which are carefully designed to elicit answers accord- ing to a number of dimensions (e.g. register, topic, and genre). For example, Table 1 shows ex- tracts from two prompts from a publicly available dataset2 that aim to elicit different genres of per- suasive/argumentative responses on different top- ics. Most previous work on AES has either ignored the differences between essays written in response to different prompts (Yannakoudakis et al., 2011) with the aim of building general AES systems, or has built prompt-specific models for each prompt independently (Chen and He, 2013; Persing and Ng, 2014). One of the problems hindering the wide-scale adoption and deployment of AES sys- tems is the dependence on prompt-specific train- ing data, i.e. …

Content truncated to 500 words