Automated Essay Scoring System for Nonnative Japanese Learners

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1250–1257 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1250 Automated Essay Scoring System for Nonnative Japanese Learners Reo Hirao, Mio Arai, Hiroki Shimanaka, Satoru Katsumata, Mamoru Komachi Tokyo Metropolitan University 6-6 Asahigaoka, Hino-city, Tokyo, Japan {hirao-reo@ed., arai-mio@ed., shimanaka-hiroki@ed., katsumata-satoru@ed., komachi@}tmu.ac.jp Abstract In this study, we created an automated essay scoring (AES) system for nonnative Japanese learners using an essay dataset with annotations for a holistic score and multiple trait scores, including content, organization, and language scores. In particular, we developed AES systems using two different approaches: a feature-based approach and a neural-network-based approach. In the former approach, we used Japanese-specific linguistic features, including character-type features such as “kanji” and “hiragana.” In the latter approach, we used two models: a long short-term memory (LSTM) model (Hochreiter and Schmidhuber, 1997) and a bidirectional encoder representations from transformers (BERT) model (Devlin et al., 2019), which achieved the highest accuracy in various natural language processing tasks in 2018. Overall, the BERT model achieved the best root mean squared error and quadratic weighted kappa scores. In addition, we analyzed the robustness of the outputs of the BERT model. We have released and shared this system to facilitate further research on AES for Japanese as a second language learners. Keywords: Learner Written Essay, Japanese, Automated Essay Scoring 1. Introduction Automated essay scoring (AES) is a task in which com- puter technology is used to evaluate written text. Humans find it difficult to evaluate a large number of essays. In this light, AES has emerged as one of the most important educa- tional applications of natural language processing. The ma- jor weakness of existing scoring systems is that they only provide a single holistic score that summarizes the quality of an essay. Such a score provides little feedback, espe- cially for a language learner. For example, when a system only returns a low holistic score, the learner cannot under- stand which aspect of the essay is inadequate without lan- guage teachers. To address this problem, some studies scored various dimensions of essay quality, such as prompt adherence (Persing and Ng, 2014), organization (Persing et al., 2010), and coherence (Miltsakaki and Kukich, 2004). Fur- thermore, some datasets are available for evaluat- ing additional dimensions of essay quality in English (Mathias and Bhattacharyya, 2018). However, only a few evaluation datasets are available for Japanese writings, and even fewer Japanese learner essay datasets are. Neverthe- less, Tanaka and Kubota overcame this lack of availabil- ity of Japanese learner datasets by creating a new dataset for Japanese learners (Tanaka and Kubota, 2016). In this dataset, they annotated a holistic score and three trait scores—content, organization, and language scores—of es- say quality on a learner’s ability. With the Japanese learner corpora annotated for multi- ple traits, we created essay scoring systems for Japanese learners using two different machine learning approaches: feature-based models and neural-network-based models. We created the feature-based models using linguistic fea- …

Content truncated to 500 words