Predicting Item Survival for Multiple Choice Questions in a High-Stakes Medical Exam

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6812–6818 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6812 Predicting Item Survival for Multiple Choice Questions in a High-stakes Medical Exam Victoria Yaneva1, Le An Ha2, Peter Baldwin1, Janet Mee1 1 - National Board of Medical Examiners, Philadelphia, USA 2 - Research Institute in Information and Language Retrieval, University of Wolverhampton, UK {vyaneva, pbaldwin, jmee}@nbme.org; l.a.ha@wlv.ac.uk Abstract One of the most resource-intensive problems in the educational testing industry relates to ensuring that newly-developed exam questions can adequately distinguish between students of high and low ability. The current practice for obtaining this information is the costly procedure of pretesting: new items are administered to test-takers and then the items that are too easy or too difficult are discarded. This paper presents the first study towards automatic prediction of an item’s probability to “survive” pretesting (item survival), focusing on human-produced MCQs for a medical exam. Survival is modeled through a number of linguistic features and embedding types, as well as features inspired by information retrieval. The approach shows promising first results for this challenging new application and for modeling the difficulty of expert-knowledge questions. Keywords: Multiple Choice Questions, Difficulty Prediction, Educational Applications 1. Introduction Large-scale testing relies on a pool of test questions, which must be replenished, updated, and expanded over time1. Writing high-quality test questions is challenging as they must satisfy certain quality standards before they can be used to score examinees. These standards are based on sta- tistical criteria and ensure that: i) items are not too easy or too difficult for the intended examinee population, and ii) the probability of success on each item is positively re- lated to overall examinee performance (Section 3.). While the exact thresholds vary, most exam programs have such a requirement. Even when item writers are well-trained and adhere to industry best practices, it has generally not been possible to identify which items will satisfy the vari- ous statistical criteria without first obtaining examinee re- sponses through pretesting. Pretesting involves embedding new items within a standard live exam and, based on the collected responses, a determination is made about whether or not a given item satisfies conditions i) and ii). Items that meet the criteria are considered to have “survived” pretest- ing and can later be used to score examinees. The propor- tion of surviving items varies across programs; however, Brennan (2006) recommends pretesting at least twice the number of items needed. While necessary, the enterprise of pretesting is costly. Scored items compete with pretest items for exam space, the scarcity of which can create a bottleneck. As a result, it is sometimes not possible to pretest as many new items as needed and some exam programs may not be able to afford pretesting at all. This problem is expected to grow with ad- vances in automatic question generation (Gierl et al., 2018), where a large amount of new questions are generated but there is …

Content truncated to 500 words