Combining Lexical and Grammatical Features to Improve Readability Measures for First and Second Language Texts

Proceedings of NAACL HLT 2007, pages 460–467, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Combining Lexical and Grammatical Features to Improve Readability Measures for First and Second Language Texts Michael J. Heilman Kevyn Collins- Thompson Jamie Callan Maxine Eskenazi Language Technologies Institute School of Computer Science Carnegie Mellon University 4502 Newell Simon Hall Pittsburgh, PA 15213-8213 {mheilman,kct,callan,max}@cs.cmu.edu Abstract This work evaluates a system that uses in- terpolated predictions of reading difficulty that are based on both vocabulary and grammatical features. The combined ap- proach is compared to individual gram- mar- and language modeling-based approaches. While the vocabulary-based language modeling approach outper- formed the grammar-based approach, grammar-based predictions can be com- bined using confidence scores with the vocabulary-based predictions to produce more accurate predictions of reading dif- ficulty for both first and second language texts. The results also indicate that gram- matical features may play a more impor- tant role in second language readability than in first language readability. 1 Introduction The REAP tutoring system (Heilman, et al. 2006), aims to provide authentic reading materials of the appropriate difficulty level, in terms of both vo- cabulary and grammar, for English as a Second Language students. An automatic measure of read- ability that incorporated both lexical and gram- matical features was thus needed. For first language (L1) learners (i.e., children learning their native tongue), reading level has been predicted using a variety of techniques, based on models of a student’s lexicon, grammatical sur- face features such as sentence length (Flesch, 1948), or combinations of such features (Schwarm and Ostendorf, 2005). It was shown by Collins- Thompson and Callan (2004) that a vocabulary- based language modeling approach was effective at predicting the readability of grades 1 to 12 of Web documents of varying length, even with high levels of noise. Prior work on first language readability by Schwarm and Ostendorf (2005) incorporated grammatical surface features such as parse tree depth and average number of verb phrases. This work combining grammatical and lexical features was promising, but it was not clear to what extent the grammatical features improved predictions. Also, discussions with L2 instructors suggest that a more detailed grammatical analysis of texts that examines features such as passive voice and various verb tenses can provide better features with which to predict reading difficulty. One goal of this work is to show that the use of pedagogically motivated grammatical features (e.g., passive voice, rather than the number of words per sen- tence) can improve readability measures based on lexical features alone. One of the differences between L1 and L2 read- ability is the timeline and processes by which first and second languages are acquired. First language acquisition begins at infancy, and the primary grammatical structures of the target language are acquired by age four in typically developing chil- 460 dren (Bates, 2003). That is, most grammar is ac- quired prior to the beginning of a child’s formal education. Therefore, most grammatical features seen at high reading levels such as high school are …

Content truncated to 500 words