A Language Modeling Approach to Predicting Reading Difficulty

Abstract We demonstrate a new research approach to the problem of predicting the reading difficulty of a text passage, by recasting readability in terms of statistical language modeling. We derive a measure based on an extension of multinomial naïve Bayes classification that combines multiple language models to estimate the most likely grade level for a given passage. The resulting classifier is not spe- cific to any particular subject and can be trained with relatively little labeled data. We perform pre- dictions for individual Web pages in English and compare our performance to widely-used semantic variables from traditional readability measures. We show that with minimal changes, the classifier may be retrained for use with French Web documents. For both English and French, the classifier main- tains consistently good correlation with labeled grade level (0.63 to 0.79) across all test sets. Some traditional semantic variables such as type-token ratio gave the best performance on commercial cal- ibrated test passages, while our language modeling approach gave better accuracy for Web documents and very short passages (less than 10 words). 1 Introduction In the course of constructing a search engine for stu- dents, we wanted a method for retrieving Web pages that were not only relevant to a student's query, but also well-matched to their reading ability. Widely-used tra- ditional readability formulas such as Flesch-Kincaid usually perform poorly in this scenario. Such formulas make certain assumptions about the text: for example, that the sample has at least 100 words and uses well- defined sentences. Neither of these assumptions need be true for Web pages or other non-traditional docu- ments. We seek a more robust technique for predicting reading difficulty that works well on a wide variety of document types. To do this, we turn to simple techniques from statis- tical language modeling. Advances in this field in the past 20 years, along with greater access to training data, make the application of such techniques to readability quite timely. While traditional formulas are based on linear regression with two or three variables, statistical language models can capture more detailed patterns of individual word usage. As we show in our evaluation, this generally results in better accuracy for Web docu- ments and very short passages (less than 10 words). Another benefit of a language modeling approach is that we obtain a probability distribution across all grade models, not just a single grade prediction. Statistical models of text rely on training data, so in Section 2 we describe our Web training corpus and note some trends that are evident in word usage. Section 3 summarizes related work on readability, focusing on existing vocabulary-based measures that can be thought of as simplified language model techniques. Section 4 defines the modified multinomial naïve Bayes model. Section 5 describes our smoothing and feature selection techniques. Section 6 evaluates our model's generaliza- tion performance, accuracy on short passages, and sen- sitivity to the amount of training data. Sections 7 and 8 discuss the evaluation results and give our observations and conclusions. …

Content truncated to 500 words