Assessment of ESL Learners’ Syntactic Competence Based on Similarity Measures

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 600–608, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Assessment of ESL Learners’ Syntactic Competence Based on Similarity Measures Su-Youn Yoon Educational Testing Service Princeton, NJ 08541 syoon@ets.org Suma Bhat Beckman Institute, Urbana, IL 61801 spbhat2@illinois.edu Abstract This study presents a novel method that measures English language learners’ syntac- tic competence towards improving automated speech scoring systems. In contrast to most previous studies which focus on the length of production units such as the mean length of clauses, we focused on capturing the differ- ences in the distribution of morpho-syntactic features or grammatical expressions across proficiency. We estimated the syntactic com- petence through the use of corpus-based NLP techniques. Assuming that the range and so- phistication of grammatical expressions can be captured by the distribution of Part-of- Speech (POS) tags, vector space models of POS tags were constructed. We use a large corpus of English learners’ responses that are classified into four proficiency levels by hu- man raters. Our proposed feature measures the similarity of a given response with the most proficient group and is then estimates the learner’s syntactic competence level. Widely outperforming the state-of-the-art measures of syntactic complexity, our method attained a significant correlation with human- rated scores. The correlation between human- rated scores and features based on manual transcription was 0.43 and the same based on ASR-hypothesis was slightly lower, 0.42. An important advantage of our method is its ro- bustness against speech recognition errors not to mention the simplicity of feature genera- tion that captures a reasonable set of learner- specific syntactic errors. 1 Introduction This study provides a novel method that measures ESL (English as a second language) learners’ com- petence in grammar usage (syntactic competence). Being interdisciplinary in nature, it shows how to combine the core findings in the ESL literature with various empirical NLP techniques for the purpose of automated scoring. Grammar usage is one of the dimensions of lan- guage ability that is assessed during non-native pro- ficiency level testing in a foreign language. Overall proficiency in the target language can be assessed by testing the abilities in various areas including flu- ency, pronunciation, and intonation; grammar and vocabulary; and discourse structure. Testing rubrics for human raters contain descriptors used for the subjective assessment of several of these features. With the recent move towards the objective assess- ment of language ability (spoken and written), it is imperative that we develop methods for quantifying these abilities and measuring them automatically. Ortega (2003) indicated that “the range of forms that surface in language production and the degree of sophistication of such forms” were two impor- tant areas in grammar usage and called the combina- tion of these two areas “syntactic complexity.” Fea- tures that measure syntactic complexity have been frequently studied in ESL literature and have been found to be highly correlated with students’ profi- ciency levels in writing. Studies in automated …

Content truncated to 500 words