A Deep Generative Approach to Native Language Identification

Proceedings of the 28th International Conference on Computational Linguistics, pages 1778–1783 Barcelona, Spain (Online), December 8-13, 2020 1778 A Deep Generative Approach to Native Language Identification Ehsan Lotfi, Ilia Markov, Walter Daelemans CLiPS Research Center University of Antwerp, Belgium {ehsan.lotfi, ilia.markov, walter.daelemans}@uantwerpen.be Abstract Native language identification (NLI) – identifying the native language (L1) of a person based on his/her writing in the second language (L2) – is useful for a variety of purposes, including mar- keting, security, and educational applications. From a traditional machine learning perspective, NLI is usually framed as a multi-class classification task, where numerous designed features are combined in order to achieve state-of-the-art results. We introduce a deep generative language modelling (LM) approach to NLI, which consists in fine-tuning a GPT-2 model separately on texts written by the authors with the same L1, and assigning a label to an unseen text based on the minimum LM loss with respect to one of these fine-tuned GPT-2 models. Our method out- performs traditional machine learning approaches and currently achieves the best results on the benchmark NLI datasets. 1 Introduction The goal of native language identification (NLI) is to automatically identify the native language (L1) of a writer based solely on his or her texts written in the second language (L2), which can inform marketing, educational, security and forensic applications, e.g., by narrowing down the list of candidate authors of a text under investigation. NLI is an interesting example of a task which is hard to perform for humans: a study of human performance in NLI (Malmasi et al., 2015) showed that they achieve around 37% accuracy, while automated NLI systems perform in the 80%–90% accuracy range, depending on the number of languages being considered, amount of data, etc. NLI is usually approached from a machine learning perspective as a multi-class classification problem of assigning class labels representing L1s to texts written in L2, where the main focus (of the traditional machine learning) is to design features that capture the systematic fingerprints of the first language in the second language writing (native language interference (Odlin, 1989)). Numerous feature types that cap- ture various aspects of the interference phenomenon have been explored for NLI: spelling errors (Koppel et al., 2005; Chen et al., 2017); lexical features, e.g., word and lemma n-grams (Jarvis et al., 2013), cognates (Markov et al., 2019), etymologically-related words (Nastase and Strapparava, 2017); syntac- tic features, e.g., context-free grammar features (Wong and Dras, 2011), Stanford parser dependency features (Tetreault et al., 2012); stylometric features, e.g., punctuation (Markov et al., 2018a), character n-gram features (Kulmizev et al., 2017); emotion-based features (Markov et al., 2018b), etc. The com- bination of such features provides the best results for NLI, as shown by the two shared tasks on the NLI task organized in 2013 (Tetreault et al., 2013) and 2017 (Malmasi et al., 2017), where the two top-ranked systems (Cimino and Dell’Orletta, 2017; Markov et al., 2017) used Support Vector Machines (SVM) with a variety of engineered features. NLI has also been …

Content truncated to 500 words