Psycholinguistic Models of Sentence Processing Improve Sentence Readability Ranking

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 958–968, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Psycholinguistic Models of Sentence Processing Improve Sentence Readability Ranking David M. Howcroft1 and Vera Demberg1,2 1 Department of Language Science and Technology 2 Department of Computer Science Saarland Informatics Campus, Saarland University Saarbr¨ucken, Germany {howcroft,vera}@coli.uni-saarland.de Abstract While previous research on readability has typically focused on document-level mea- sures, recent work in areas such as nat- ural language generation has pointed out the need of sentence-level readability mea- sures. Much of psycholinguistics has fo- cused for many years on processing mea- sures that provide difficulty estimates on a word-by-word basis. However, these psycholinguistic measures have not yet been tested on sentence readability rank- ing tasks. In this paper, we use four psycholinguistic measures: idea density, surprisal, integration cost, and embedding depth to test whether these features are predictive of readability levels. We find that psycholinguistic features significantly improve performance by up to 3 percent- age points over a standard document-level readability metric baseline. 1 Introduction Previous work on readability has classified or ranked texts based on document-level measures such as word length, sentence length, number of different phrasal categories & parse tree depth (Pe- tersen, 2007), and discourse coherence (Graesser et al., 2004), inter alia. However, not all appli- cations that need readability ratings deal with long documents. For many applications in text simplifi- cation, computer-aided language learning (CALL) systems, authorship tools, translation, and infor- mation retrieval, sentence-level readability metrics are direly needed. For instance, an automatic text simplification system must begin by asking which portions of a text need to be simplified. To this end, a measure that can assign ratings on a sentence-by-sentence level can help target simplification only to those sentences which need it most, and such measures also serve to confirm that the resulting ‘simplified’ sentence is in fact simpler than the original sen- tence. Similarly, CALL and other pedagogical systems will benefit if it is possible to predict which por- tions of a text will be harder for students. Au- thorship tools can offer more specific editorial ad- vice when they know why individual sentences can cause difficulties for readers. Translation tools can aim to preserve not just meaning but also the approximate difficulty of the sentences they are translating or use a sentence-level difficulty metric to target output that is easier to understand. Fur- thermore, information retrieval systems also bene- fit when they can return not merely relevant texts, but also texts appropriate to the reading level of the user. Recently there has been an increased interest in sentential models of text difficulty in the auto- matic text simplification and summarization com- munities in particular (Vajjala and Meurers, 2014; Macdonald and Siddharthan, 2016). One area that has produced a lot of research on sentence level processing difficulty is psycholin- guistics. Over the past three decades, a number of theories of human sentence processing (i.e. …

Content truncated to 500 words