Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text

Proceedings of Recent Advances in Natural Language Processing, pages 846–855 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_097 846 Towards the application of calibrated Transformers to the unsupervised estimation of question difficulty from text Ekaterina Loginova Ghent University, Ghent, Belgium ekaterina.loginova@ugent.be Luca Benedetto Politecnico di Milano, Milan, Italy luca.benedetto@polimi.it Dries Benoit Ghent University, Ghent, Belgium dries.benoit@ugent.be Paolo Cremonesi Politecnico di Milano, Milan, Italy paolo.cremonesi@polimi.it Abstract Being able to accurately perform Question Dif- ficulty Estimation (QDE) can improve the ac- curacy of students’ assessment and better their learning experience. Traditional approaches to QDE are either subjective or introduce a long delay before new questions can be used to assess students. Thus, recent work pro- posed machine learning-based approaches to overcome these limitations. They use ques- tions of known difficulty to train models ca- pable of inferring the difficulty of questions from their text. Once trained, they can be used to perform QDE of newly created questions. Existing approaches employ supervised mod- els which are domain-dependent and require a large dataset of questions of known difficulty for training. Therefore, they cannot be used if such a dataset is not available (e.g. for new courses on an e-learning platform). In this work, we experiment with the possibility of performing QDE from text in an unsupervised manner. Specifically, we use the uncertainty of calibrated question answering models as a proxy of human-perceived difficulty. Our ex- periments show promising results, suggesting that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time. 1 Introduction Question Difficulty Estimation (QDE), also known as “question calibration”, is a crucial task in edu- cation. In Computerized Adaptive Testing (Linden et al., 2000), for instance, students are shown ques- tions that are suitable for their skill level. When a question is miscalibrated (i.e. its difficulty has been erroneously estimated), it can be either too hard or too easy for a student, which would nega- tively affect their learning outcome (Wang, 2014). If the questions are too hard, students might get frustrated and lose motivation; if they are too easy, students are not adequately challenged (Papousek et al., 2016). In either case, their learning experi- ence is worse than if the questions were of appro- priate difficulty, which, especially in the context of large-scale online courses, gives rise to the methods of automated difficulty estimation. Traditionally, QDE is performed either i) manu- ally (Attali et al., 2014) or ii) with pretesting (Lane et al., 2015). Manual calibration involves one (or more) human experts labelling the question by se- lecting a numerical value representing its difficulty, a method that is subjective and not scalable. Mean- while, pretesting involves deploying the new ques- tion in an exam as if it was a standard exam ques- tion, but without using it to assess students and without telling them that there is a question un- der pretesting. The other questions of the exams are then used to actually assess the students, and their answers help to calibrate the question under pretesting. This …

Content truncated to 500 words