Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1485 - 1495 July 10-15, 2022 ©2022 Association for Computational Linguistics Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays ♣Rahul Kumar, ♠Sandeep Mathias, ♢Sriparna Saha, ♡ Pushpak Bhattacharyya ♣Department of Computer Science and Engineering, IIT Kanpur ♠Department of Computer Science and Engineering, Presidency University, Bangalore ♢Department of Computer Science and Engineering, IIT Patna ♡Department of Computer Science and Engineering, IIT Bombay ♣rahul@iitk.ac.in, ♠sandeepalbert@presidencyuniversity.in, ♢sriparna@iitp.ac.in, ♡pb@cse.iitb.ac.in Abstract Most research in the area of automatic essay grading (AEG) is geared towards scoring the essay holistically while there has also been lit- tle work done on scoring individual essay traits. In this paper, we describe a way to score essays using a multi-task learning (MTL) approach, where scoring the essay holistically is the pri- mary task, and scoring the essay traits is the auxiliary task. We compare our results with a single-task learning (STL) approach, using both LSTMs and BiLSTMs. To find out which traits work best for different types of essays, we conduct ablation tests for each of the essay traits. We also report the runtime and number of training parameters for each system. We find that MTL-based BiLSTM system gives the best results for scoring the essay holistically, as well as performing well on scoring the essay traits. The MTL systems also give a speed-up of be- tween 2.30 to 3.70 times the speed of the STL system, when it comes to scoring the essay and all the traits. 1 Introduction An essay is a piece of text that is written in re- sponse to a topic, called a prompt (Mathias and Bhattacharyya, 2020). Qualitative evaluation of the essay consumes a lot of time and resources. Hence, in 1966, Page proposed a method of automatically scoring essays using computers (Page, 1966), giv- ing rise to the domain of Automatic Essay Grading. Essay traits are different aspects of the essay that can aid in explaining the score assigned to the es- say. Examples of essay traits include content (how much information is present in the essay) (Page, 1966), organization (how well the essay is struc- tured) (Persing et al., 2010), style (how well written the essay is) (Page, 1966), prompt adherence (how much the essay stays on topic for the essay prompt) (Persing and Ng, 2014), etc. Most of the research work done in the field of AEG is geared toward scoring the essay holistically, rather than studying the importance of essay traits in the overall essay score. In this paper, we ask the question: “Can we use information learnt from scoring essay traits to score an essay holistically?” In our paper, not only do we score essays holisti- cally, but we also describe how to score essay traits simultaneously in a multi-task learning framework. Scoring essay traits is essential as it could help in explaining why the essay was scored the way it was, as well as …

Content truncated to 500 words