DT-QDC: A Dataset for Question Comprehension in Online Test

Proceedings of the 28th International Conference on Computational Linguistics, pages 6470–6480 Barcelona, Spain (Online), December 8-13, 2020 6470 DT-QDC: A Dataset for Question Comprehension in Online Test Sijin Wu1, Yujiu Yang 1 (�), Nicholas Yung2 , Zhengchen Shen3, Zeyang Lei4 1Tsinghua Shenzhen International Graduate School, Tsinghua University 2Bielcrystal Datalab 3Institute for Network Sciences and Cyberspace, Tsinghua University 4Baidu Inc {wusj18, szc18}@mails.tsinghua.edu.cn, yang.yujiu@sz.tsinghua.edu.cn nicholas.yung@bielcrystal.com.hk, leizeyang@baidu.com Abstract As education switches from the traditional classroom environment to online education and as- sessment, teachers could not follow students’ learning behavior closely which leads to inaccurate assessment of students’ knowledge. A well-defined method to precisely measure difficulty of questions is critical as it can guide teachers in assessing students’ performance and help provide customized attention to students. In this paper, we explore the concept of question difficulty and share our new Chinese Driving Test Question Difficulty Comprehension (DT-QDC) dataset. Different from the existing question datasets, we mainly design for question evaluation and com- prehension in online testing, so each question has enriched attributes and difficulty label. Addi- tional attributes such as keywords, chapter, and question type would allow models to understand questions more precisely. We proposed the Ordinal Regression Multi-Source BERT (ORMS- BERT) model to assess the difficulty of the question, which outperforms different baselines by 6.77% on F1-score and 15.92% on MAE, 28.26% on MSE on the DT-QDC dataset, laying the foundation for the question difficulty comprehension task. 1 Introduction Intelligent education systems are heavily studied and investigated since it can generate great value both academically and commercially, especially when the COVID-19 pandemics is prevailing. A well- developed educational assistance system can facilitate students to grasp the learning progress and cus- tomize personalized learning approaches. It improves the whole learning experience, enhances students’ initiative, and results in better performance in assessments. One key component of an intelligent education system is to assess the level of understanding of stu- dents, and asking questions is the most intuitive way of accomplishing that. However, the most online quiz system treats questions equally without factoring in the difference in difficulty. Some researcher (Sonkar et al., 2020) argued that we should not treat all questions equivalently, be- cause questions exhibit significant variations in difficulty and discrimination (Embretson and Reise, 2013). Pardos et al. (Pardos and Heffernan, 2011) once introduced the modeling of the problem dif- ficulty in the knowledge tracing task, but they only used the guess-and-slip parameters related to the question, without paying attention to the nature of the question itself. For every new question, they must use the corresponding user answer data to estimate its difficulty. In this paper, we go one step further to investigate methods that perform question difficulty com- prehension using various attributes of the question. Question understanding is one of the key com- ponent in machine reading comprehension, which is viewed as the sign of machine understand nature language (Nakanishi et al., 2018). Although question comprehension is not studied as a special task, it is positively related to the difficulty of the question. …

Content truncated to 500 words