Machine Learning–Driven Language Assessment

Machine Learning–Driven Language Assessment Burr Settles and Geoffrey T. LaFlair Duolingo Pittsburgh, PA USA {burr,geoff}@duolingo.com Masato Hagiwara∗ Octanove Labs Seattle, WA USA masato@octanove.com Abstract We describe a method for rapidly creating lan- guage proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural lan- guage processing to induce proficiency scales based on a given standard, and then use linguis- tic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores alignsignificantlywithotherhigh-stakes English assessments. Furthermore, our approach pro- duces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements. 1 Introduction Language proficiency testing is an increasingly important part of global society. The need to dem- onstrate language skills—often through standard- ized testing—is now required in many situations for access to higher education, immigration, and employment opportunities. However, standard- ized tests are cumbersome to create and main- tain. Lane et al. (2016) and the Standards for Educational and Psychological Testing (AERA et al., 2014) describe many of the procedures and requirements for planning, creating, revis- ing, administering, analyzing, and reporting on high-stakes tests and their development. In practice, test items are often first written by subject matter experts, and then ‘‘pilot tested’’ with a large number of human subjects for psy- ∗ Research conducted at Duolingo. chometric analysis. This labor-intensive process often restricts the number of items that can feasibly be created, which in turn poses a threat to security: Items may be copied and leaked, or simply used too often (Cau, 2015; Dudley et al., 2016). Security can be enhanced through computer- adaptive testing (CAT), by which a subset of items are administered in a personalized way (based on examinees’ performance on previous items). Because the item sequences are essentially unique for each session, there is no single test form to obtain and circulate (Wainer, 2000), but these security benefits only hold if the item bank is large enough to reduce item exposure (Way, 1998). This further increases the burden on item writers, and also requires significantly more item pilot testing. For the case of language assessment, we tackle both of these development bottlenecks using machine learning (ML) and natural language processing (NLP). In particular, we propose the use of test item formats that can be automatically created, graded, and psychometrically analyzed using ML/NLP techniques. This solves the ‘‘cold start’’ problem in language test development, by relaxing manual item creation requirements and alleviating the need for human pilot testing altogether. In the pages that follow, we first summarize the important concepts from language testing and psychometrics (§2), and then describe our ML/NLP methods to learn proficiency scales for both words (§3) and long-form passages (§4). We then present evidence for the validity, reliability, and security of our approach …

Content truncated to 500 words