Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory

Proceedings of the 29th International Conference on Computational Linguistics, pages 2917–2926 October 12–17, 2022. 2917 Analytic Automated Essay Scoring based on Deep Neural Networks Integrating Multidimensional Item Response Theory Takumi Shibata The University of Electro-Communications shibata@ai.lab.uec.ac.jp Masaki Uto The University of Electro-Communications uto@ai.lab.uec.ac.jp Abstract Essay exams have been attracting attention as a way of measuring the higher-order abilities of examinees, but they have two major draw- backs in that grading them is expensive and raises questions about fairness. As an approach to overcome these problems, automated essay scoring (AES) is in increasing need. Many AES models based on deep neural networks have been proposed in recent years and have achieved high accuracy, but most of these mod- els are designed to predict only a single overall score. However, to provide detailed feedback in practical situations, we often require not only the overall score but also analytic scores corre- sponding to various aspects of the essay. Sev- eral neural AES models that can predict both the analytic scores and the overall score have also been proposed for this very purpose. How- ever, conventional models are designed to have complex neural architectures for each analytic score, which makes interpreting the score pre- diction difficult. To improve the interpretabil- ity of the prediction while maintaining scor- ing accuracy, we propose a new neural model for automated analytic scoring that integrates a multidimensional item response theory model, which is a popular psychometric model. 1 Introduction Rapid changes in society in recent years have led to an increased need for cultivating and assessing not only knowledge and skills but also practical abilities, such as expression skills, logical thinking, and creativity (Erguvan and Aksu Dunya, 2020; Uto, 2021a). Essay exams are one of the test for- mats that aim to evaluate these abilities, and conse- quently, they have been used in various educational and assessment settings (Erguvan and Aksu Dunya, 2020; Hussein et al., 2019). However, essay exams have two considerable drawbacks in the time and monetary costs required to grade them (Taghipour and Ng, 2016). Furthermore, it is difficult to ensure consistently fair and reliable evaluation due to sub- jective influences on the part of the rater (Uto and Ueno, 2020; Saal et al., 1980). Automated Essay Scoring (AES) has been attracting attention as a method for resolving these difficulties (Dong and Zhang, 2016; Taghipour and Ng, 2016). Conventional AES systems can be broadly clas- sified into two categories (Hussein et al., 2019): those that take a feature-engineering approach and those that take a neural approach. The feature- engineering approach, which has traditionally been the greater used of the two, utilizes a statistical or machine learning model with pre-defined hand- crafted features (e.g. Attali and Burstein, 2006; Chen and He, 2013; Phandi et al., 2015; Dascalu et al., 2017; Hastings et al., 2018; Yao et al., 2019). The neural approach, on the other hand, which has become popular recently, uses deep neural networks to extract features automatically from texts (e.g. Alikaniotis et al., 2016; Taghipour …

Content truncated to 500 words