Happy Are Those Who Grade without Seeing: A Multi-Task Learning Approach to Grade Essays Using Gaze Behaviour

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 858–872 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 858 Happy Are Those Who Grade without Seeing: A Multi-Task Learning Approach to Grade Essays Using Gaze Behaviour Sandeep Mathias♠, Rudra Murthy♠,♢, Diptesh Kanojia♠,♣, Abhijit Mishra♢, Pushpak Bhattacharyya♠ ♠ Department of Computer Science, Indian Institute of Technology, Bombay ♢ IBM Research, India ♣ IITB-Monash Research Academy {sam,rudra,diptesh,pb}@cse.iitb.ac.in, abhijitmishra.530@gmail.com Abstract The gaze behaviour of a reader is helpful in solving several NLP tasks such as automatic essay grading. However, collecting gaze be- haviour from readers is costly in terms of time and money. In this paper, we propose a way to improve automatic essay grading using gaze behaviour, which is learnt at run time using a multi-task learning framework. To demon- strate the efficacy of this multi-task learning based approach to automatic essay grading, we collect gaze behaviour for 48 essays across 4 essay sets, and learn gaze behaviour for the rest of the essays, numbering over 7000 es- says. Using the learnt gaze behaviour, we can achieve a statistically significant improvement in performance over the state-of-the-art system for the essay sets where we have gaze data. We also achieve a statistically significant improve- ment for 4 other essay sets, numbering about 6000 essays, where we have no gaze behaviour data available. Our approach establishes that learning gaze behaviour improves automatic essay grading. 1 Introduction Collecting a reader’s psychological input can be very beneficial to a number of Natural Language Processing (NLP) tasks, like complexity (Mishra et al., 2017; Gonz´alez-Gardu˜no and Søgaard, 2017), sentence simplification (Klerke et al., 2016), text understanding (Mishra et al., 2016), text quality (Mathias et al., 2018), parsing (Hale et al., 2018), etc. This psychological information can be ex- tracted using devices like eye-trackers, and elec- troencephalogram (EEG) machines. However, one of the challenges in using reader’s information in- volves collecting the psycholinguistic data itself. In this paper, we choose the task of automatic essay grading and show how we can predict the score that a human rater would give using both text and learnt gaze behaviour. An essay is a piece of text, written in response to a topic, called a prompt. Automatic essay grading is assigning a score to the essay using a machine. An essay set is a set of essays written in response to the same prompt. Multi-task learning (Caruana, 1998) is a machine learning paradigm where we utilize auxiliary tasks to aid in solving a primary task. This is done by exploiting similarities between the primary task and the auxiliary tasks. Scoring the essay is the primary task and learning gaze behaviour is the auxiliary task. Using gaze behaviour for a very small number of essays (less than 0.7% of the essays in an es- say set), we see an improvement in predicting the overall score of the essays. We also use our gaze behaviour …

Content truncated to 500 words