Reproducing Monolingual, Multilingual and Cross-Lingual CEFR Predictions

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5595–5602 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5595 Reproducing Monolingual, Multilingual and Cross-Lingual CEFR Predictions Yves Bestgen LAST — UCLouvain Place Cardinal Mercier, 10, 1348 Louvain-la-Neuve Belgium yves.bestgen@uclouvain.be Abstract This study aims to reproduce the research of Vajjala and Rama (2018) which showed that it is possible to predict the quality of a text written by learners of a given language by means of a model built on the basis of texts written by learners of another language. These authors also pointed out that POStag and dependency n-grams were significantly more effective than text length and global linguistic indices frequently used for this kind of task. The analyses performed show that some important points of their code did not correspond to the explanations given in the paper. These analyses confirm the possibility to use syntactic n-gram features in cross-lingual experiments to categorize texts according to their CEFR level (Common European Framework of Reference for Languages). However, text length and some classical indexes of readability are much more effective in the monolingual and the multilingual experiments than what Vajjala and Rama concluded and are even the best performing features when the cross-lingual task is seen as a regression problem. This study emphasized the importance for reproducibility of setting explicitly the reading order of the instances when using a K-fold CV procedure and, more generally, the need to properly randomize these instances before. It also evaluates a two-step procedure to determine the degree of statistical significance of the differences observed in a K-fold cross-validation schema and argues against the use of a Bonferroni-type correction in this context. Keywords: automated essay scoring, k-fold cross-validation, statistical significance test 1. Introduction Automated essay scoring (AES) is one of the main applica- tion areas of natural language processing in education and teaching as evidenced by the commercial systems available on the market for many years such as the Intelligent Es- say Assessor, eRater or IntelliMetric (Shermis and Ham- mer, 2013). Used most often as a second assessment in addition to a human evaluator, these systems facilitate the deployment of standardized tests by reducing the necessary human resources (Williamson et al., 2012). Such systems are particularly useful in the field of foreign language learn- ing where being able to write high-quality texts is of most importance (Bestgen, 2019). The state-of-the-art approach in this area relies on linguis- tic clues potentially correlated with the text quality such as the presence of typographical, orthographic and syntac- tic errors, detected among others on the basis of n-grams, but also the length of the text, the presence of very short sentences, the absence of an introduction or a conclusion (Williamson et al., 2012). Training materials, consisting of at least several hundred texts evaluated by human experts, are used to establish the most effective model for predicting these human evaluations. This model is then applied to new texts. Most of the works …

Content truncated to 500 words