Predicting Text Readability from Scrolling Interactions

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 380–390 November 10–11, 2021. ©2021 Association for Computational Linguistics 380 Predicting Text Readability from Scrolling Interactions Sian Gooding Dept of Computer Science and Technology University of Cambridge shg36@cam.ac.uk Yevgeni Berzak MIT BCS berzak@mit.edu Tony Mak Google tonymak@google.com Matt Sharifi Google mns@google.com Abstract Judging the readability of text has many impor- tant applications, for instance when perform- ing text simplification or when sourcing read- ing material for language learners. In this pa- per, we present a 518 participant study which investigates how scrolling behaviour relates to the readability of English texts. We make our dataset publicly available and show that (1) there are statistically significant differences in the way readers interact with text depending on the text level, (2) such measures can be used to predict the readability of text, and (3) the background of a reader impacts their reading interactions and the factors contributing to text difficulty.1 1 Introduction There are multiple attributes of a written text that impact how difficult it is to read. This concept is formally known as the readability of text, where readability is defined as the sum total of elements within a text that impact a reader’s understanding, reading speed and level of interest (Dale and Chall, 1949). Many factors can influence the readability of text, such as the lexical and syntactic complexity, level of conceptual difficulty and style of writing (Xia et al., 2016). For instance, Figure 1 presents an example sentence (b) that has been rewritten to a more readable format shown in (a).2 Automatically measuring the readability of text has many useful applications. For example, it is used in text simplification (Aluisio et al., 2010), when sourcing reading material for language learn- ers (Xia et al., 2016), in ranking search engine con- tent for dyslexic users (Morris et al., 2018) and to ensure critical consumer information is delivered at an appropriate level (Zou et al., 2019). How- ever, current approaches for measuring readability 1Our dataset is available at https://github.com/ siangooding/readability_scroll 2Example from the OneStopEnglish dataset by Vajjala and Luˇci´c (2018). (a) A new scientific study says that global warming might make temperatures rise more than people think. (b) Temperature rises resulting from unchecked climate change will be at the severe end of those projected, according to a new scientific study. Figure 1: Example of two sentences (a) and (b) with differing levels of readability. Sentence (b) has a more sophisticated syntactic structure and advanced vocabu- lary than (a). rely exclusively on linguistic features which do not account for the subjective needs of readers. As a result, traditional readability formulas perform poorly in modeling adult judgements of textual complexity (Crossley et al., 2017). Furthermore, traditional readability approaches do not work well for online content (Collins- Thompson, 2013). This is due to systems being highly sensitive to noise; requiring well formed sentences and performing poorly on short passages (Collins-Thompson, 2014). Linguistically driven techniques are also language specific and require sophisticated models to …

Content truncated to 500 words