Predicting proficiency levels in learner writings by transferring a linguistic complexity model from expert-written coursebooks

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2101–2111, Osaka, Japan, December 11-17 2016. Predicting proficiency levels in learner writings by transferring a linguistic complexity model from expert-written coursebooks Ildik´o Pil´an, Elena Volodina Spr˚akbanken, University of Gothenburg Sweden ildiko.pilan@svenska.gu.se elena.volodina@svenska.gu.se Torsten Zesch Language Technology Lab University of Duisburg-Essen Germany torsten.zesch@uni-due.de Abstract The lack of a sufficient amount of data tailored for a task is a well-recognized problem for many statistical NLP methods. In this paper, we explore whether data sparsity can be successfully tackled when classifying language proficiency levels in the domain of learner-written output texts. We aim at overcoming data sparsity by incorporating knowledge in the trained model from another domain consisting of input texts written by teaching professionals for learners. We compare different domain adaptation techniques and find that a weighted combination of the two types of data performs best, which can even rival systems based on considerably larger amounts of in-domain data. Moreover, we show that normalizing errors in learners’ texts can substantially improve classification when in-domain data with annotated proficiency levels is not available. 1 Introduction Data sparsity is a recognized problem in many machine learning based NLP approaches since the creation of data specifically collected and annotated for a certain task or language is time-consuming and costly. Previous attempts to overcome data sparsity include transferring knowledge between different types of data through the application of models from languages and tasks where sufficient data exists to the ones where data is unavailable or sparse (Daum´e III and Marcu, 2006). A common case of such a transfer learning scenario is domain adaptation, where training and test data belong to different domains (e.g. text genres) referred to as source domain and target domain respectively. In our experiments, we aim at exploring the plausibility of domain adaptation as a strategy for over- coming data sparsity in the context of foreign and second language (L2) learning. More specifically, we operationalize domain as the type of text involved in the language learning process: on the one hand, texts from coursebooks intended for L2 learners (referred to as L2 input texts in this paper), and on the other hand, essays created by learners (L2 output texts). Our goal is to predict L2 language development stages in terms of linguistic complexity in the latter category, i.e. learner-produced texts. These stages are commonly referred to as proficiency levels in second language acquisition and language testing. Lev- els range from ‘absolute beginner’ to ‘advanced language user’ with increasing linguistic complexity as learners progress with the levels. A scale of such levels, very influential both in Europe and outside, is the CEFR – Common European Framework of Reference for Languages (Council of Europe, 2001). In previous work, NLP methods have been successfully applied to both assessing proficiency levels in L2 input texts collected from coursebooks and output texts written by learners (see section 2). How- ever, the two text types have always been considered separately, while we argue that there …

Content truncated to 500 words