Automatic evaluation of surface coherence in L2 texts in Czech

Automatic evaluation of surface coherence in L2 texts in Czech Kateřina Rysová, Magdaléna Rysová, Jiří Mírovský Charles University in Prague, Czech Republic Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {rysova|magdalena.rysova|mirovsky}@ufal.mff.cuni.cz Abstract We introduce possibilities of automatic evaluation of surface text coherence (cohesion) in texts written by learners of Czech during certified exams for non-native speakers. On the basis of a corpus analysis, we focus on finding and describing relevant distinctive features for automatic detection of A1–C1 levels (established by CEFR – the Common European Framework of Reference for Languages) in terms of surface text coherence. The CEFR levels are evaluated by human assessors and we try to reach this assessment automatically by using several discourse features like frequency and diversity of discourse connectives, density of discourse relations etc. We present experiments with various features using two machine learning algorithms. Our results of automatic evaluation of CEFR coherence/cohesion marks (compared to human assessment) achieved 73.2% success rate for the detection of A1–C1 levels and 74.9% for the detection of A2–B2 levels. 1 Introduction Our research is carried out on texts written during the international language examinations provided by the Test Centre of the Institute of Language and Preparatory Studies at the Charles University in Prague in line with the high ALTE (Association of Language Testers The 2016 Conference on Computational Linguistics and Speech Processing ROCLING 2016, pp. 214-228  The Association for Computational Linguistics and Chinese Language Processing 214 in Europe) standards. Such type of examination is required by Czech universities (the needed CEFR level is usually B2) or often also by employers and the exam is compulsory for foreigners to be granted permanent residence in the Czech Republic (the required CEFR level is A1) or state citizenship (the required CEFR level is B1).1 Therefore, it is of great importance to assess these examinations as objectively as possible and according to uniform criteria. This is rather difficult because the writing samples are evaluated manually by human assessors (although according to the uniform rating grid) who naturally bring to the evalu- ation a subjective human factor. In the present paper, we aim at finding several objective criteria (concerning surface text coherence) for distinguishing the individual CEFR levels automatically. Specifically, we carry out a research on surface text coherence concerning various discourse phenomena (like the use and frequency of connectives etc.) and we test the possibility of their automatic monitoring and evaluating. The results of our research will become a part of a software application that will serve as a tool for objective assessment of surface text coherence, i.e. for automatic division of submitted writing samples into the suitable CEFR levels in the coherence/cohesion category. 2 Previous Research There are many studies and projects dealing with automatic evaluation of various language phenomena especially for English. Many of them focus on grammatical aspects of language (e.g. on automatic evaluation of grammatical accuracy, detection of grammatical errors etc. – see [1]; [2] or [3]). On the other hand, only few of …

Content truncated to 500 words