Revisiting Readability: A Unified Framework for Predicting Text Quality

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Revisiting Readability: A Unified Framework for Predicting Text Quality Emily Pitler Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA epitler@seas.upenn.edu Ani Nenkova Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA nenkova@seas.upenn.edu Abstract We combine lexical, syntactic, and discourse features to produce a highly predictive model of human readers’ judgments of text readabil- ity. This is the first study to take into ac- count such a variety of linguistic factors and the first to empirically demonstrate that dis- course relations are strongly associated with the perceived quality of text. We show that various surface metrics generally expected to be related to readability are not very good pre- dictors of readability judgments in our Wall Street Journal corpus. We also establish that readability predictors behave differently de- pending on the task: predicting text readabil- ity or ranking the readability. Our experi- ments indicate that discourse relations are the one class of features that exhibits robustness across these two tasks. 1 Introduction The quest for a precise definition of text quality— pinpointing the factors that make text flow and easy to read—has a long history and tradition. Way back in 1944 Robert Gunning Associates was set up, of- fering newspapers, magazines and business firms consultations on clear writing (Gunning, 1952). In education, teaching good writing technique and grading student writing has always been of key importance (Spandel, 2004; Attali and Burstein, 2006). Linguists have also studied various aspects of text flow, with cohesion-building devices in English (Halliday and Hasan, 1976), rhetorical structure the- ory (Mann and Thompson, 1988) and centering the- ory (Grosz et al., 1995) among the most influential contributions. Still, we do not have unified computational mod- els that capture the interplay between various as- pects of readability. Most studies focus on a sin- gle factor contributing to readability for a given in- tended audience. The use of rare words or technical terminology for example can make text difficult to read for certain audience types (Collins-Thompson and Callan, 2004; Schwarm and Ostendorf, 2005; Elhadad and Sutaria, 2007). Syntactic complexity is associated with delayed processing time in un- derstanding (Gibson, 1998) and is another factor that can decrease readability. Text organization (dis- course structure), topic development (entity coher- ence) and the form of referring expressions also de- termine readability. But we know little about the rel- ative importance of each factor and how they com- bine in determining perceived text quality. In our work we use texts from the Wall Street Journal intended for an educated adult audience to analyze readability factors including vocabulary, syntax, cohesion, entity coherence and discourse. We study the association between these features and reader assigned readability ratings, showing that dis- course and vocabulary are the factors most strongly linked to text quality. In the easier task of text qual- ity ranking, entity coherence and syntax features also become significant …

Content truncated to 500 words