Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12456–12470 July 9-14, 2023 ©2023 Association for Computational Linguistics Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation Learning Zhiwei Jiang∗†, Tianyi Gao∗, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, Qing Gu State Key Laboratory for Novel Software Technology, Nanjing University, China jzw@nju.edu.cn, mf21330021@smail.nju.edu.cn, yafeng@nju.edu.cn {mf1933061,huayu.yh,chengzf}@smail.nju.edu.cn, guq@nju.edu.cn Abstract Automated Essay Scoring (AES) aims to score essays written in response to specific prompts. Many AES models have been proposed, but most of them are either prompt-specific or prompt-adaptive and cannot generalize well on "unseen" prompts. This work focuses on improving the generalization ability of AES models from the perspective of domain gen- eralization, where the data of target prompts cannot be accessed during training. Specifi- cally, we propose a prompt-aware neural AES model to extract comprehensive representa- tion for essay scoring, including both prompt- invariant and prompt-specific features. To im- prove the generalization of representation, we further propose a novel disentangled represen- tation learning framework. In this framework, a contrastive norm-angular alignment strategy and a counterfactual self-training strategy are designed to disentangle the prompt-invariant information and prompt-specific information in representation. Extensive experimental re- sults on datasets of both ASAP and TOEFL11 demonstrate the effectiveness of our method under the domain generalization setting. 1 Introduction Automated Essay Scoring (AES), which aims to score essays written for specific prompts, is helpful in reducing the burden of scoring staff in various writing tests (Ke and Ng, 2019). Over the past few years, supervised deep learning has achieved re- markable success on the prompt-specific AES task (Taghipour and Ng, 2016; Farag et al., 2018; Tay et al., 2018), which assumes that the training and test data are from the same prompt. However, in many real-world scenarios, the training and test data often come from different prompts, which leads to a performance degradation of prompt- specific AES model on the out-of-distribution tar- * Both authors contributed equally to this research. † Corresponding author. Prompt Generalization AES Model Unseen Prompt Source Prompts P1 Essays P2 Essays 1 2 3 1 2 3 Essays P3 Train Test Prompt Adaptation AES Model Target Prompt Source Prompts P1 Essays P2 Essays 1 2 3 1 2 3 Essays P3 Train Test Prompt Specific AES Model Test Set Training Set P1 Essays 1 2 3 Essays P1 Train Test 1 2 3 Figure 1: Comparison among prompt specific, prompt adaptation, and prompt generalization settings. get prompt (Dong and Zhang, 2016; Cozma et al., 2018). Many researchers have tried to adapt the AES model from source prompts to the target prompt, with limited labeled data (Cozma et al., 2018; Cao et al., 2020) or only unlabeled data (Jin et al., 2018) in target prompt. Despite their success, they need to access the data of target prompts during training and may fail to work when the target prompt is unavailable during training. To this end, in this paper, we focus on the prompt generalization …

Content truncated to 500 words