Aggregating Multiple Heuristic Signals as Supervision for Unsupervised Automated Essay Scoring

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13999–14013 July 9-14, 2023 ©2023 Association for Computational Linguistics Aggregating Multiple Heuristic Signals as Supervision for Unsupervised Automated Essay Scoring Cong Wang, Zhiwei Jiang∗, Yafeng Yin, Zifeng Cheng, Shiping Ge, Qing Gu State Key Laboratory for Novel Software Technology, Nanjing University, China cw@smail.nju.edu.cn, {jzw,yafeng}@nju.edu.cn, {chengzf,shipingge}@smail.nju.edu.cn, guq@nju.edu.cn Abstract Automated Essay Scoring (AES) aims to eval- uate the quality score for input essays. In this work, we propose a novel unsupervised AES approach ULRA, which does not require groundtruth scores of essays for training. The core idea of our ULRA is to use multiple heuris- tic quality signals as the pseudo-groundtruth, and then train a neural AES model by learning from the aggregation of these quality signals. To aggregate these inconsistent quality signals into a unified supervision, we view the AES task as a ranking problem, and design a spe- cial Deep Pairwise Rank Aggregation (DPRA) loss for training. In the DPRA loss, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural AES model in a pairwise way to disentangle the cascade effect among partial- order pairs. Experiments on eight prompts of ASPA dataset show that ULRA achieves the state-of-the-art performance compared with previous unsupervised methods in terms of both transductive and inductive settings. Fur- ther, our approach achieves comparable per- formance with many existing domain-adapted supervised models, showing the effectiveness of ULRA. The code is available at https: //github.com/tenvence/ulra. 1 Introduction Automated Essay Scoring (AES) that aims to score the writing quality of essays without human in- tervention, is an important application of natural language processing in education. State-of-the-art AES models are typically trained in a supervised way with large labeled corpora, comprising essays and their groundtruth quality scores (Cozma et al., 2018; Ke and Ng, 2019; Kumar et al., 2022; Wang et al., 2022). However, collecting labeled essays is time-consuming and labor-intensive, especially for essays written specific to new prompts and when there is no professional scoring staff available. ∗ Corresponding author. Unsupervised AES can get rid of the require- ment of groundtruth scores for training, and thus has significant potential in both scientific research and practical applications. Its importance can be summarized in three key aspects: 1) Unsupervised AES models can handle special scenarios that lack labeling resource, such as the absence of profes- sional scoring staff, the need for rapid essay scoring without timely labeled data, or the cold start scor- ing of an AES system without historical labeled data; 2) Unsupervised AES models can serve as pseudo-label generators or validators for essay scor- ing based on semi-supervised learning, few-shot learning, or transfer learning; 3) In practical writ- ing tests, unsupervised AES models can rapidly provide a preliminary decision-making basis for scoring staff prior to scoring. Early work tackles unsupervised AES by using the clustering method (Chen et al., 2010). To solve the problem of unordered clusters (i.e., cannot map clusters …

Content truncated to 500 words