MALT-IT2: A New Resource to Measure Text Difficulty in Light of CEFR Levels for Italian L2 Learning

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7204–7211 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7204 MALT-IT2: A New Resource to Measure Text Difficulty in light of CEFR levels for Italian L2 learning Luciana Forti, Giuliana Grego Bolli, Filippo Santarelli, Valentino Santucci, Stefania Spina Department of Humanities and Social Sciences University for Foreigners of Perugia Piazza G. Spitella, 3 - 06123 Perugia (Italy) {luciana.forti, giuliana.grego, filippo.santarelli, valentino.santucci, stefania.spina}@unistrapg.it Abstract This paper presents a new resource for automatically assessing text difficulty in the context of Italian as a second or foreign language learning and teaching. It is called MALT-IT2, and it automatically classifies inputted texts according to the CEFR level they are more likely to belong to. After an introduction to the field of automatic text difficulty assessment, and an overview of previous related work, we describe the rationale of the project, the corpus and computational system it is based on. Experiments were conducted in order to investigate the reliability of the system. The results show that the system is able to obtain a good prediction accuracy, while a further analysis was conducted in order to identify the categories of features which mostly influenced the predictions. Keywords: text difficulty, Italian L2, assessment 1. Introduction Determining the appropriateness of a text in relation to CEFR levels has a key role in the context of second or foreign language teaching and assessment. The suitabil- ity of a text for a certain learner group is generally estab- lished on the basis of its thematic content, as it needs to be adequate to the learners’ personal motivation for learn- ing, and to its linguistic content, as it needs to be in line with their level of proficiency. In regards to the latter cri- terion, evaluations of the difficulty of a text are generally conducted subjectively, both when a text needs to be cho- sen as a component of a language test, and when it needs to be chosen for classroom use. The formal and quantitative characteristics of a text, in particular, have a major role in determining the comprehensibility of that text, as they will impose specific cognitive demands upon the reader when approaching the text (Bachman and Palmer, 2010; Purpura, 2014). However, in the absence of an objective difficulty measure, not only do subjective judgments lead to poten- tial discrepancies between different individual evaluations made by teachers and/or test developers (Franc¸ois et al., 2014), but it also makes it impossible to gain insight into a large number of quantitative features that have an impact on the difficulty of a text, and on the cognitive load involved. A number of research projects have been conducted with the aim to automatically assess the difficulty of a text. Most involve English, both with the aim of simplify- ing administrative texts and of second/foreign language learning purposes, though recent years have seen a rise in projects involving languages other than English such French, Swedish, Dutch, German …

Content truncated to 500 words