Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1560 Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking Ruosong Yang†‡, Jiannong Cao†, Zhiyuan Wen†, Youzheng Wu‡, Xiaodong He‡ †Department of Computing, The Hong Kong Polytechnic University ‡JD AI Research †{csryang, csjcao, cszwen}@comp.polyu.edu.hk ‡{wuyouzheng1, xiaodong.he}@jd.com Abstract Automated Essay Scoring (AES) is a critical text regression task that automatically assigns scores to essays based on their writing qual- ity. Recently, the performance of sentence prediction tasks has been largely improved by using Pre-trained Language Models via fus- ing representations from different layers, con- structing an auxiliary sentence, using multi- task learning, etc. However, to solve the AES task, previous works utilize shallow neural net- works to learn essay representations and con- strain calculated scores with regression loss or ranking loss, respectively. Since shallow neu- ral networks trained on limited samples show poor performance to capture deep semantic of texts. And without an accurate scoring func- tion, ranking loss and regression loss mea- sures two different aspects of the calculated scores. To improve AES’s performance, we find a new way to fine-tune pre-trained lan- guage models with multiple losses of the same task. In this paper, we propose to utilize a pre- trained language model to learn text represen- tations first. With scores calculated from the representations, mean square error loss and the batch-wise ListNet loss with dynamic weights constrain the scores simultaneously. We uti- lize Quadratic Weighted Kappa to evaluate our model on the Automated Student Assessment Prize dataset. Our model outperforms not only state-of-the-art neural models near 3 percent but also the latest statistic model. Especially on the two narrative prompts, our model per- forms much better than all other state-of-the- art models. 1 Introduction Automated Essay Scoring (AES) automatically evaluates the writing quality of essays. Essay as- signments evaluation costs lots of time. Besides, the same instructor scoring the same essay at dif- ferent times may assign different scores (intra-rater variation), different raters scoring the same essay may assign different scores (inter-rater variation) (Smolentzov, 2013). To alleviate teachers’ bur- den and avoid intra-rater variation, as well as inter- rater variation, AES is necessary and essential. An early AES system, e-rater (Chodorow and Burstein, 2004), has been used to score TOEFL writings. Recently, large pre-trained language models, such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), etc. have shown the extraordinary ability of representation and generalization. These models have gained bet- ter performance in lots of downstream tasks such as text classification and regression. There are many new approaches to fine-tune pre-trained language models. Sun et al. (2019a) proposed to construct an auxiliary sentence to solve aspect-based senti- ment classification tasks. Cohan et al. (2019) added extra separate tokens to obtain representations of each sentence to solve sequential sentence classifi- cation tasks. Sun et al. (2019b) summarized several fine-tuning methods, including fusing text …

Content truncated to 500 words