Evaluating the Knowledge Dependency of Questions

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10512 - 10526 December 7-11, 2022 ©2022 Association for Computational Linguistics Evaluating the Knowledge Dependency of Questions Hyeongdon Moon∗ † 1 Yoonseok Yang∗ † 2 Jamin Shin† Hangyeol Yu1 Seunghyun Lee1 Myeongho Jeong1 Juneyoung Park1 Minsam Kim† 1 Seungtaek Choi† 1 1Riiid AI Research 2UC Berkeley hyeongdon.mun@riiid.co, yoonseok@berkeley.edu, jayshin.nlp@gmail.com, {minsam.kim, seungtaek.choi}@riiid.co Abstract The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assess- ment significantly. However, existing evalu- ation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disre- gard their educational value. They fail to eval- uate the MCQ’s ability to assess the student’s knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge De- pendent Answerability (KDA), which mea- sures the MCQ’s answerability given knowl- edge of the target fact. Specifically, we first show how to measure KDA based on stu- dent responses from a human survey. Then, we propose two automatic evaluation metrics, KDAdisc and KDAcont, that approximate KDA by leveraging pre-trained language mod- els to imitate students’ problem-solving behav- ior. Through our human studies, we show that KDAdisc and KDAcont have strong corre- lations with both (1) KDA and (2) usabil- ity in an actual classroom setting, labeled by experts. Furthermore, when combined with n- gram based similarity metrics, KDAdisc and KDAcont are shown to have a strong predic- tive power for various expert-labeled MCQ quality measures. 1 1 Introduction Multiple-Choice Question (MCQ), comprised of a question stem, an answer, and a set of distrac- tors, is one of the most widely used student assess- ment tools. Since manually generating exam-style questions is a complex, labor-intensive process ∗ Equal Contribution. † Corresponding authors. Work was done while all cor- responding authors were at Riiid AI Research. Jamin Shin is currently at NAVER AI Lab. 1 Our code is released here: https://github.com/ riiid/question-score (a) Bad question (too easy) (b) Good question Figure 1: Problem formulation. A multiple-choice ques- tion designed to test the knowledge of a target fact must satisfy the following criterion: a question has to be an- swered by only a student who knows the fact, not a student who doesn’t. that requires training, experience, and resources, automatic question generation (AQG) techniques were introduced (Kurdi et al., 2020). If automatic MCQ generation methods develop to a level that re- quires only minor adjustments by educators, it can meet the real-world demand for creating a massive amount of MCQ sets in seconds. Despite the importance of AQG in educational purposes, the task did not receive much atten- tion and was not actively introduced in the ed- ucation field due to the limitation of evaluation methods. Previous AQG works mostly evaluate 10512 their methods based on how similar the gener- ated questions/distractors are to the gold ques- …

Content truncated to 500 words