Are Cohesive Features Relevant for Text Readability Evaluation?

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 987–997, Osaka, Japan, December 11-17 2016. Are Cohesive Features Relevant for Text Readability Evaluation? Amalia Todirascu FDT, LiLPa Universit´e de Strasbourg todiras@unistra.fr Thomas Franc¸ois Post-doc FNRS IL&C, CENTAL UCLouvain thomas.francois@uclouvain.be Delphine Bernhard FDT, LiLPa Universit´e de Strasbourg dbernhard@unistra.fr N´uria Gala LIF-CNRS, Aix-Marseille Universit´e nuria.gala@univ-amu.fr Anne-Laure Ligozat LIMSI-CNRS, Orsay annlor@limsi.fr Abstract This paper investigates the effectiveness of 65 cohesion-based variables that are commonly used in the literature as predictive features to assess text readability. We evaluate the efficiency of these variables across narrative and informative texts intended for an audience of L2 French learners. In our experiments, we use a French corpus that has been both manually and automatically an- notated as regards to co-reference and anaphoric chains. The efficiency of the 65 variables for readability is analyzed through a correlational analysis and some modelling experiments. 1 Introduction Since the 1920’s, various readability formulae have been designed to match texts with the reading skills of specific readers. The most famous of these formulas, such as Flesch’s (1948) or Dale and Chall’s (1948) are typical of what are called “classic” formulas. They rely on a few lexico-syntactic character- istics (e.g., the average number of words per sentence or the average number of syllables per word) to estimate the reading difficulty of a text. This strategy worked to some extent, but, from the late 70’s onward, classic formulae have been seriously criticised. Zakaluk and Samuels (1988, 124) thus said: “A basic limitation of readability formulas is that they ignore such critical text factors as cohesiveness and macrolevel organization”. Studies in readability from this period stressed the importance of higher textual dimensions, focus- ing on inference load (Kemper, 1983), conceptual density (Kintsch and Vipond, 1979), or organisational aspects (Meyer, 1982). As a result, the classic lexico-syntactic features were disregarded for years. How- ever, Miller and Kintsch (1980) soon noticed that including lexico-syntactic features in their cognitive readability formulas improved performance. Chall and Dale (1995, 111) had a more mixed opinion, ar- guing that variables based on higher textual dimensions “discriminate better among materials requiring greater maturity in reading ability”, while classic lexico-syntactic variables work better to discriminate at lower levels of difficulty. Recently, taking advantage of the opportunities offered by Natural Language Processing (NLP) tech- niques, readability studies have tried to leverage the semantic and discursive properties of texts to better model text difficulty (Pitler and Nenkova, 2008; Feng et al., 2009). Among those high-level dimensions that have attracted substantial attention are the level of cohesion and coherence of texts. Although psy- cholinguistic experiments have shown that a higher level of cohesion and coherence between a pair of related sentences decreases their reading time (Kintsch et al., 1975; Mason and Just, 2004), the added value of these textual dimensions for readability models (compared to traditional features) remains un- clear, as it will be covered in more details in Section 2. This is why this paper aims at further investigating the …

Content truncated to 500 words