Multi-Stage Pre-training for Automated Chinese Essay Scoring

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6723–6733, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6723 Multi-Stage Pre-training for Automated Chinese Essay Scoring Wei Song1, Kai Zhang1, Ruiji Fu2,3, Lizhen Liu1, Ting Liu4, Miaomiao Cheng1 1College of Information Engineering and Academy for Multidisciplinary Studies, Capital Normal University, Beijing, China 2State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China 3iFLYTEK AI Research (Hebei), Langfang, China 4Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China {wsong, kzhang, liz liu7480, B365}@cnu.edu.cn, rjfu@iflytek.com, tliu@ir.hit.edu.cn Abstract This paper proposes a pre-training based au- tomated Chinese essay scoring method. The method involves three components: weakly supervised pre-training, supervised cross- prompt fine-tuning and supervised target- prompt fine-tuning. An essay scorer is first pre- trained on a large essay dataset covering di- verse topics and with coarse ratings, i.e., good and poor, which are used as a kind of weak su- pervision. The pre-trained essay scorer would be further fine-tuned on previously rated es- says from existing prompts, which have the same score range with the target prompt and provide extra supervision. At last, the scorer is fine-tuned on the target-prompt training data. The evaluation on four prompts shows that this method can improve a state-of-the-art neural essay scorer in terms of effectiveness and do- main adaptation ability, while in-depth analy- sis also reveals its limitations. 1 Introduction Automated essay scoring (AES) is an important educational application of natural language pro- cessing (NLP) (Page, 1966). AES aims to automat- ically judge the quality of student essays, which can reduce teachers’ burden on essay scoring and provide fast feedback to students. AES is usually viewed as a supervised learning problem. Traditionally, AES systems are based on hand-crafted surface-level features (Larkey, 1998; Attali and Burstein, 2006; Chen and He, 2013; Phandi et al., 2015). Recently, neural network based representation learning has been applied and achieved superior performance compared with tra- ditional methods (Taghipour and Ng, 2016; Cum- mins et al., 2016; Alikaniotis et al., 2016; Dong and Zhang, 2016; Dong et al., 2017; Tay et al., 2018). Most of the proposed methods, no matter the feature based or the representation learning based ones, work in an in-domain setting that is to train a scorer for a specific prompt based on a set of example essays for this prompt, and use this scorer to rate more essays from the same prompt. This manner usually requires many rated examples to get acceptable performance. Although cross-domain transferable essay scor- ing has gained more attention (Phandi et al., 2015; Jin et al., 2018), the progress is still limited. The possible reason may be that the available corpora for essay scoring usually cover narrow topics on a small scale, and the topics, scoring criteria, and score ranges of different prompts often vary. Since an AES system should have the ability to appreci- ate or criticize essays, supervised pre-training is necessary. Intuitively, if a reader has read many rated essays from different prompts, she …

Content truncated to 500 words