Automated Prediction of Examinee Proficiency from Short-Answer Questions

Proceedings of the 28th International Conference on Computational Linguistics, pages 893–903 Barcelona, Spain (Online), December 8-13, 2020 893 Automated Prediction of Examinee Proficiency from Short-Answer Questions Le An Ha1 Victoria Yaneva1,2 Polina Harik2 Ravi Pandian2 Amy Morales2 Brian Clauser2 1Research Group in Computational Linguistics, University of Wolverhampton, UK ha.l.a@wlv.ac.uk 2National Board of Medical Examiners, Philadelphia, USA {vyaneva, pharik, rpandian, amorales, bclauser}@nbme.org Abstract This paper brings together approaches from the fields of NLP and psychometric measurement to address the problem of predicting examinee proficiency from responses to short-answer questions (SAQs). While previous approaches train on manually labeled data to predict the human ratings assigned to SAQ responses, the approach presented here models examinee proficiency directly and does not require manually labeled data to train on. We use data from a large medical exam where experimental SAQ items are embedded alongside 106 scored multiple-choice questions (MCQs). First, the latent trait of examinee proficiency is measured using the scored MCQs and then a model is trained on the experimental SAQ responses as input, aiming to predict proficiency as its target variable. The predicted value is then used as a “score” for the SAQ response and evaluated in terms of its contribution to the precision of proficiency estimation. 1 Introduction The automated scoring of Short Answer Questions (SAQs) has been a longstanding research area in both psychometric measurement and NLP applications. Typically, a SAQ consists of a description of a prob- lem followed by an open question which requires some form of a free short response by the test-taker. An example of a SAQ testing medical knowledge is presented in Table 1. The task at hand is to score the re- sponses given by the test-takers, either manually or automatically. What makes the problem challenging is the fact that the answers, even the correct ones, tend to vary a lot in their expression and level of detail. The educational measurement literature has shown that the manual scoring of SAQs requires significant resources and often suffers from low rater agreement (Section 2.1). In NLP, the automated scoring of SAQ responses is still far from being solved and requires large amounts of expensive expert-rated data (Section 2.2). The lack of affordable and reliable SAQ scoring solutions perpetuates the assessment field’s reliance on less informative but easy to score formats like Multiple Choice Questions (MCQs). Despite the fact that the automated scoring of SAQs is a vibrant research area in both psychometrics and educational NLP, there are almost no interdisciplinary efforts that capitalize on the strengths of both fields (Section 2). In psychometrics, the majority of the effort is focused on assessing the reliability, validity and fairness of the scoring procedures for measuring examinee proficiency, with low exposure to state-of-the-art NLP techniques. By the same token, NLP studies utilize sophisticated language tech- nology but do not evaluate the scoring systems in a way that answers the main question of “Does this scoring method help with the more precise measurement of examinee proficiency?”. This paper presents an interdisciplinary …

Content truncated to 500 words