Native Tongues, Lost and Found: Resources and Empirical Evaluations in Native Language Identification

Native Tongues, Lost and Found: Resources and Empirical Evaluations in Native Language Identification Joel T ETREAU LT 1 Daniel BLANCHARD1 Aoi f e CAHI LL1 Martin CHODOROW 2 (1) Educational Testing Service, Rosedale Road, Princeton, NJ 08541, USA (2) Hunter College and the Graduate Center, City University of New York, New York, NY 10065, USA tetreaul@gmail.com, dblanchard@ets.org, acahill@ets.org, martin.chodorow@hunter.cuny.edu ABSTRACT In this paper we present work on the task of Native Language Identification (NLI). We present an alternative corpus to the ICLE which has been used in most work up until now. We believe that our corpus, TOEFL11, is more suitable for the task of NLI and will allow researchers to better compare systems and results. We show that many of the features that have been commonly used in this task generalize to new and larger corpora. In addition, we examine possible ways of increasing current system performance (e.g., additional features and feature combination methods), and achieve overall state-of-the-art results (accuracy of 90.1%) on the ICLE corpus using an ensemble classifier that includes previously examined features and a novel feature (n-gram language models). We also show that training on a large corpus and testing on a smaller one works well, but not vice versa. Finally, we show that system performance varies across proficiency scores. KEYWORDS: Native Language Identification, Text Classification, Corpora. 1 Introduction One growing NLP field is that of Native Language Identification (NLI), which is the task of automatically identifying a speaker’s first language based solely on the speaker’s writing in another language. NLI can be useful for a number of applications. Native language is often used as a feature in machine learning approaches to authorship profiling (Estival et al., 2007), which is frequently used in forensic linguistics. NLI can also be used in educational settings to provide more targeted feedback to language learners about their errors (Chang et al., 2008; Dahlmeier and Ng, 2011; Rozovskaya and Roth, 2011). It is well known that speakers of different languages make different kinds of errors when learning a language (Swan and Smith, 2001). For example, a French speaker learning English might write sentence (1), which contains a verb tense error. On the other hand, a Japanese speaker learning English might make the verb tense error shown in (2). A writing tutor system which can detect the native language of the learner will be able to tailor the feedback about the error and contrast it with common properties of the learner’s language. (1) She knows that she hasn’t achieve it completely. (2) They also said to have great curiosity. There has been a great deal of work on NLI in recent years. The methods employed have ranged from some combination of lexical, part-of-speech and n-gram features (Koppel et al., 2005), to syntactic features (Wong and Dras, 2011) including Tree Substitution Grammars (TSGs) (Swanson and Charniak, 2012), to topic models (Wong et al., 2011). Despite these research efforts, it has been somewhat hard to compare different approaches for a number of reasons. The …

Content truncated to 500 words