Large-Scale Native Language Identification with Cross-Corpus Evaluation

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1403–1409, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Large-scale Native Language Identification with Cross-Corpus Evaluation Shervin Malmasi Centre for Language Technology Macquarie University Sydney, NSW, Australia shervin.malmasi@mq.edu.au Mark Dras Centre for Language Technology Macquarie University Sydney, NSW, Australia mark.dras@mq.edu.au Abstract We present a large-scale Native Language Identification (NLI) experiment on new data, with a focus on cross-corpus evaluation to identify corpus- and genre-independent lan- guage transfer features. We test a new corpus and show it is comparable to other NLI cor- pora and suitable for this task. Cross-corpus evaluation on two large corpora achieves good accuracy and evidences the existence of reli- able language transfer features, but lower per- formance also suggests that NLI models are not completely portable across corpora. Fi- nally, we present a brief case study of fea- tures distinguishing Japanese learners’ En- glish writing, demonstrating the presence of cross-corpus and cross-genre language trans- fer features that are highly applicable to SLA and ESL research. 1 Introduction Native Language Identification, the task of deter- mining the native language (L1) of an author based on a second language (L2) text, has received much attention recently. Much of this is motivated by Sec- ond Language Acquisition (SLA) as NLI, often ac- complished via machine learning methods, can be used to study language transfer effects. Most NLI research hitherto has focused on identi- fying linguistic phenomena that can capture trans- fer effects, with little effort towards interpreting discriminant features. Some researchers have now shifted their focus to developing data-driven meth- ods for the automatic extraction and ranking of lin- guistic features that distinguish specific L1s (Swan- son and Charniak, 2014). Such methods could be used not only to confirm existing SLA hypotheses, but also to create new ones. This hypothesis formulation is an inherently difficult problem requiring copious amounts of data. Contrary to this requirement, researchers have long noted the paucity of suitable corpora1 for this task (Brooke and Hirst, 2011). This is one of the research issues addressed by this work. Furthermore, deriving SLA hypotheses from a single corpus may not be entirely useful for SLA research. Many variables like genre and topic are constant within a corpus, restricting the validity of such cross-validation studies to those dimensions. An alternative, potentially more helpful approach, is to identify transfer features that reliably distin- guish an L1 across multiple corpora of differing gen- res and domains. A cross-corpus methodology may be a more promising avenue to finding features that generalize to diverse text sources, but requires addi- tional large corpora. It is also a more realistic ap- proach, and one we pursue in this work. Accordingly, the aims of the present work are to: (1) test a large new corpus suitable for NLI, (2) perform within-corpus evaluation with a compara- tive analysis against equivalent corpora, (3) perform cross-corpus evaluation to determine the efficiency of corpus independent features and (4) …

Content truncated to 500 words