Predicting the Proficiency Level of Nonnative Hebrew Authors

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5356–5365 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5356 Predicting the Proficiency Level of Nonnative Hebrew Authors Isabelle Nguyen, Shuly Wintner Humboldt University, Berlin, Germany; University of Haifa, Israel isabelle.nguyen@pm.me, shuly@cs.haifa.ac.il Abstract We present classifiers that can accurately predict the proficiency level of nonnative Hebrew learners. This is important for practical (mainly educational) applications, but the endeavor also sheds light on the features that support the classification, thereby improving our understanding of learner language in general, and transfer effects from Arabic, French, and Russian on nonnative Hebrew in particular. Keywords: Learner corpora, Hebrew, Proficiency 1. Introduction The language of nonnative speakers (L2) differs from native language (L1), and learners take time to reach a level of proficiency that is comparable with native language. Assessing the proficiency level of nonna- tive speakers is essential for tasks such as determining university admittance, adaptation of language learning tools, tailoring foreign language training for students, etc. We show that the proficiency level of nonnative He- brew speakers (defined here technically, as scores in language aptitude tests) can be accurately predicted from the essays they author. On a recently-released cor- pus of Hebrew essays authored by prospective students with three different mother tongues (Arabic, Russian, and French), we present simple classifiers that can dis- tinguish between native and nonnative speakers, iden- tify the native language of the author, and predict the essay’s proficiency score with accuracy that does not differ much from that of human graders. We then pro- vide a detailed analysis of the features that enable the classification, thereby deriving insight into the charac- teristics of second language, transfer effects from the native language, and in general, a better understanding of bilingualism. After discussing related work, we describe the exper- imental setup for this work in Section 3. The experi- ments’ results are presented in Section 4, followed by an analysis in Section 5. We conclude with suggestions for future work. 2. Related work Assessing the proficiency level of nonnative speakers is an important task that has great benefits for educational applications, and automating it is therefore an ongoing endeavor, typically aided by learner corpora (L¨udeling et al., 2008; Blanchard et al., 2013; Wisniewski et al., 2013). Various measures have been proposed for assessing for- eign language proficiency (Scarborough, 1990; Ortega, 2003; Attali and Burstein, 2005; Bult´e and Housen, 2012; Zesch et al., 2015; Weiss, 2017, and many more). However, calculating these measures is complex, ex- pensive and most importantly, does not always achieve good results. Consequently, methods based on text classification have been widely adopted in recent years. Cross- ley et al. (2012) analyzed 100 English essays au- thored by speakers of 19 different L1s. They found that the strongest predictors of proficiency level were word imagability, word frequency, lexical diversity, and word familiarity. On this small dataset, their in- dices identified the proficiency level of authors (on a 2-class scale) with 70% …

Content truncated to 500 words