Assessing Language Proficiency from Eye Movements in Reading

Proceedings of NAACL-HLT 2018, pages 1986–1996 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Assessing Language Proficiency from Eye Movements in Reading Yevgeni Berzak MIT BCS berzak@mit.edu Boris Katz MIT CSAIL boris@mit.edu Roger Levy MIT BCS rplevy@mit.edu Abstract We present a novel approach for determining learners’ second language proficiency which utilizes behavioral traces of eye movements during reading. Our approach provides stand- alone eyetracking based English proficiency scores which reflect the extent to which the learner’s gaze patterns in reading are similar to those of native English speakers. We show that our scores correlate strongly with standard- ized English proficiency tests. We also demon- strate that gaze information can be used to ac- curately predict the outcomes of such tests. Our approach yields the strongest performance when the test taker is presented with a suite of sentences for which we have eyetracking data from other readers. However, it remains effec- tive even using eyetracking with sentences for which eye movement data have not been previ- ously collected. By deriving proficiency as an automatic byproduct of eye movements during ordinary reading, our approach offers a poten- tially valuable new tool for second language proficiency assessment. More broadly, our re- sults open the door to future methods for infer- ring reader characteristics from the behavioral traces of reading. 1 Introduction It is currently estimated that over 1.5 billion peo- ple are learning English as a Second Language (ESL) worldwide. Their learning progress is com- monly evaluated with classroom tests prepared by language instructors, quizzes in language learn- ing software such as Duolingo and Rosetta Stone, and by official standardized language proficiency tests such as TOEFL, IELTS, MET and others. In “high stakes” scenarios, official language profi- ciency tests are the de-facto standards for language assessment; they are accepted by educational and professional institutions, and are taken by millions of language learners every year (for example, in 2016 over three million people took the IELTS test (IELTS, 2017)). These tests probe language profi- ciency based on performance on various linguistic tasks, including grammar and vocabulary exams, reading and listening comprehension questions, as well as essay writing and speaking assignments. Despite their ubiquity, traditional approaches to language proficiency testing have several draw- backs. First, such tests are typically prepared man- ually and require extensive resources for test de- velopment. Moreover, their validity can be un- dermined by test specific training, prior knowl- edge of the evaluation mechanisms (Powers et al., 2002), as well as plain cheating via unauthorized access to test materials. Further, the utilized test- ing and evaluation methodologies vary across dif- ferent tests, and test materials are in most cases inaccessible to the research community. Perhaps most crucially, the reliance of these tests on the end products of linguistic tasks makes it challeng- ing to study learners’ language processing patterns and the difficulties they encounter in real time. In this work we propose a novel methodol- ogy for language proficiency assessment which marks a significant departure from …

Content truncated to 500 words