When Shallow is Good Enough: Automatic Assessment of Conceptual Text Complexity using Shallow Semantic Features

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1414–1422 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1414 When Shallow is Good Enough: Automatic Assessment of Conceptual Text Complexity using Shallow Semantic Features Sanja ˇStajner1 and Ioana Hulpus¸2 1ReadableAI, Cologne, Germany 2Data and Web Science Group, University of Mannheim, Germany sanja.stajner@readableai.com, ioana@informatik.uni-mannheim.de Abstract According to psycholinguistic studies, the complexity of concepts used in a text and the relations between mentioned concepts play the most important role in text understanding and maintaining reader’s interest. However, the classical approaches to automatic assessment of text complexity, and their commercial applications, take into consideration mainly syntactic and lexical complexity. Recently, we introduced the task of automatic assessment of conceptual text complexity, proposing a set of graph-based deep semantic features using DBpedia as a proxy to human knowledge. Given that such graphs can be noisy, incomplete, and computationally expensive to deal with, in this paper, we propose the use of textual features and shallow semantic features that only require entity linking. We compare the results obtained with new features with those of the state-of-the-art deep semantic features on two tasks: (1) pairwise comparison of two versions of the same text; and (2) five-level classification of texts. We find that the shallow features achieve state-of-the-art results on both tasks, significantly outperforming performances of the deep semantic features on the five-level classification task. In- terestingly, the combination of the shallow and deep semantic features lead to a significant improvement of the performances on that task. Keywords: conceptual complexity, semantics, text classification 1. Introduction In this digitalized era, readers are overwhelmed with the amount of informational texts freely available online. Al- though seemingly so easy to reach, many of them are too complex for an average reader, and even more for strug- gling readers, e.g. non-native speakers, people with low literacy, or people with any kind of reading or cognitive impairments (such as dyslexia, aphasia, autism, Down’s syndrome, etc.). To mitigate this problem, many guide- lines have been proposed and made available advising on how to make texts easier to understand for everyone, re- garding both their presentation (in terms of font styles, sizes, and colors, background colors, etc.) and their textual characteristics (vocabulary, syntactic structures, and overall text structures), e.g. Web Content Accessibility Guidelines (WCAG)1, ‘Make it Simple’ guidelines (Freyhoff et al., 1998), Federal Plain Language Guidelines2. At the same time, many readability metrics have been proposed to man- ually or automatically assess the reading level necessary to understand a text, mostly focusing on superficial lexical and syntactic features, and rarely on more semantic-based fea- tures (DuBay, 2004). Most of the proposed readability met- rics do not assess deeper levels of text processing, such as inference making and the use of world knowledge and dis- course structure necessary to comprehend the text (Arf´e et al., 2017). According to the model of reading comprehension pro- posed by Kintsch and van Dijk (1978), the reader needs to understand both …

Content truncated to 500 words