Learning to Predict Readability using Diverse Linguistic Features

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 546–554, Beijing, August 2010 Learning to Predict Readability using Diverse Linguistic Features Rohit J. Kate1 Xiaoqiang Luo2 Siddharth Patwardhan2 Martin Franz2 Radu Florian2 Raymond J. Mooney1 Salim Roukos2 Chris Welty2 1Department of Computer Science The University of Texas at Austin {rjkate,mooney}@cs.utexas.edu 2IBM Watson Research Center {xiaoluo,spatward,franzm,raduf,roukos,welty}@us.ibm.com Abstract In this paper we consider the problem of building a system to predict readability of natural-language documents. Our sys- tem is trained using diverse features based on syntax and language models which are generally indicative of readability. The experimental results on a dataset of docu- ments from a mix of genres show that the predictions of the learned system are more accurate than the predictions of naive hu- man judges when compared against the predictions of linguistically-trained expert human judges. The experiments also com- pare the performances of different learn- ing algorithms and different types of fea- ture sets when used for predicting read- ability. 1 Introduction An important aspect of a document is whether it is easily processed and understood by a human reader as intended by its writer, this is termed as the document’s readability. Readability in- volves many aspects including grammaticality, conciseness, clarity, and lack of ambiguity. Teach- ers, journalists, editors, and other professionals routinely make judgements on the readability of documents. We explore the task of learning to automatically judge the readability of natural- language documents. In a variety of applications it would be useful to be able to automate readability judgements. For example, the results of a web-search can be or- dered taking into account the readability of the retrieved documents thus improving user satisfac- tion. Readability judgements can also be used for automatically grading essays, selecting in- structional reading materials, etc. If documents are generated by machines, such as summariza- tion or machine translation systems, then they are prone to be less readable. In such cases, a read- ability measure can be used to automatically fil- ter out documents which have poor readability. Even when the intended consumers of text are machines, for example, information extraction or knowledge extraction systems, a readability mea- sure can be used to filter out documents of poor readability so that the machine readers will not ex- tract incorrect information because of ambiguity or lack of clarity in the documents. As part of the DARPA Machine Reading Pro- gram (MRP), an evaluation was designed and con- ducted for the task of rating documents for read- ability. In this evaluation, 540 documents were rated for readability by both experts and novice human subjects. Systems were evaluated based on whether they were able to match expert readabil- ity ratings better than novice raters. Our system learns to match expert readability ratings by em- ploying regression over a set of diverse linguistic features that were deemed potentially relevant to readability. Our results demonstrate that a rich combination of features from syntactic parsers, language models, as well as lexical statistics all contribute to accurately …

Content truncated to 500 words