Language Proficiency Scoring

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5624–5630 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5624 Language Proficiency Scoring Cristina Arhiliuc, Jelena Mitrovi´c, Michael Granitzer Faculty of Computer Science and Mathematics University of Passau, Germany arhili01@ads.uni-passau.de, jelena.mitrovic@uni-passau.de, michael.granitzer@uni-passau.de Abstract The Common European Framework of Reference (CEFR) provides generic guidelines for evaluation of language proficiency. Neverthe- less, for automated proficiency classification systems, different approaches for different languages are proposed. Our paper evaluates and extends the results of an approach to Automatic Essay Scoring proposed as a part of the REPROLANG 2020 challenge. We provide a comparison between our results and the ones from the original paper, and we also include experiments on a new corpus for the English language. Our results are lower than the expected when using the same approach, and the system does not scale well with the added English corpus. Keywords: AES, Automatisation, Language Proficiency 1. Introduction In the world of globalization and internationalization be- ing multilingual allows for more business opportunities. This drives more individuals to learn additional languages, which in turn increases the number of language exams such as TOEFL and IELTS for English, TCF, DELF and DALF for French, telc, TestDaF, and Goethe-Institut for German, taken a few times every year. The Common European Framework of Reference (CEFR) offers a generalized scoring system of language proficiency of learners that consists of 6 levels independent of the lan- guage: A1, A2, B1, B2, C1 and C2. Automated Essay Scoring (AES) represents the task of automatically assess- ing texts written by learners using natural language process- ing tools. The verification and validation of a new AES ap- proach are part of the REPROLANG 2020 challenge1 along with many other research topics in the area of natural lan- guage processing. The goal of our work is to reproduce the results published in the original, candidate paper (Vajjala and Rama, 2018), that explores the possibility of a multilingual approach of classifying texts and to extend their approach with a new corpus. A multilingual model represents a model trained on multiple languages and capable of classifying texts in multiple languages. In our paper, we discuss several issues: • Would building a multilingual model instead of a monolingual one have a great impact on the prediction metrics? • Which features could improve the prediction metrics for multilingual models? What is their impact on the monolingual model? • What are the limitations of the current model and how can it be improved? The remainder of our paper is organized as follows. Section 2. gives a short overview of the State of the Art research on AES approaches. A short description of the used corpora 1REPROLANG 2020, https://www.clarin.eu/event/2020/reprolang- 2020, last accessed on July 21, 2019 is presented in section 3., followed by the methodology ap- plied in this paper in section 4. Section 5. shows the results of reproducing the original paper’s experiments. Further- more, section 6. describes the cross-lingual experiments. …

Content truncated to 500 words