Predicting the Relative Difficulty of Single Sentences With and Without Surrounding Context

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1871–1881, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Predicting the Relative Difficulty of Single Sentences With and Without Surrounding Context Elliot Schumacher Carnegie Mellon University eschumac@cs.cmu.edu Maxine Eskenazi Carnegie Mellon University max@cs.cmu.edu Gwen Frishkoff University of Oregon gfrishkoff@gmail.com Kevyn Collins-Thompson University of Michigan kevynct@umich.edu Abstract The problem of accurately predicting rela- tive reading difficulty across a set of sen- tences arises in a number of important natu- ral language applications, such as finding and curating effective usage examples for intelli- gent language tutoring systems. Yet while significant research has explored document- and passage-level reading difficulty, the spe- cial challenges involved in assessing aspects of readability for single sentences have re- ceived much less attention, particularly when considering the role of surrounding passages. We introduce and evaluate a novel approach for estimating the relative reading difficulty of a set of sentences, with and without surround- ing context. Using different sets of lexical and grammatical features, we explore models for predicting pairwise relative difficulty using lo- gistic regression, and examine rankings gener- ated by aggregating pairwise difficulty labels using a Bayesian rating system to form a final ranking. We also compare rankings derived for sentences assessed with and without con- text, and find that contextual features can help predict differences in relative difficulty judg- ments across these two conditions. 1 Introduction The reading difficulty, or readability, of a text is an estimate of linguistic complexity and is typically based on lexical and syntactic features, such as text length, word frequency, and grammatical complex- ity (Collins-Thompson and Callan, 2004; Schwarm and Ostendorf, 2005; Kidwell et al., 2011; Kanungo and Orr, 2009). Such estimates are often expressed as age- or grade-level measures and are useful for a range of educational and research applications. For example, instructors often wish to select stories or books that are appropriately matched to student grade level. Many measures have been designed to calculate readability at the document level (e.g., for web pages, articles, or books) (Collins-Thompson and Callan, 2004; Schwarm and Ostendorf, 2005), as well as the paragraph or passage level (Kidwell et al., 2011; Kanungo and Orr, 2009). However, much less work has attempted to characterize the readabil- ity of single sentences (Pil´an et al., 2014). This problem is challenging because single sentences provide less data than is typically required for re- liable estimates, particularly for measures that rely on aggregate statistics. The absence of reliable single-sentence estimates points to a gap in natural language processing (NLP) research. Single sentences are used in a variety of experimental and NLP applications: for example, in studies of reading comprehension. Because read- ability estimates have been shown to predict a sub- stantial portion of variance in comprehension of dif- ferent texts, it would be useful to have measures of single-sentence readability. Thus, one aim of the current study was to estimate the relative readabil- ity of single sentences with a high degree of accu- racy. …

Content truncated to 500 words