LILI: A Simple Language Independent Approach for Language Identification

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1211–1219, Osaka, Japan, December 11-17 2016. LILI: A Simple Language Independent Approach for Language Identification Mohamed Al-Badrashiny and Mona Diab Department of Computer Science, The George Washington University {badrashiny,mtdiab}@gwu.edu Abstract We introduce a generic Language Independent Framework for Linguistic Code Switch Point De- tection. The system uses the word length, character level (1, 2, 3, 4, and 5)-grams and word level unigram language models to train a conditional random fields (CRF) model for classifying input words into various languages. We test our proposed framework and compare it to the state-of-the- art published systems on standard data sets from several language pairs: English-Spanish, Nepali- English, English-Hindi, Arabizi (Refers to Arabic written using the Latin/Roman script)-English, Arabic-Engari (Refers to English written using Arabic script), Modern Standard Arabic(MSA)- Egyptian, Levantine-MSA, Gulf-MSA, one more English-Spanish, and one more MSA-EGY. The overall weighted average F-score of each language pair are 96.4%, 97.3%, 98.0%, 97.0%, 98.9%, 86.3%, 88.2%, 90.6%, 95.2%, and 85.0% respectively. The results show that our ap- proach despite its simplicity, either outperforms or performs at comparable levels to state-of-the- art published systems. 1 Introduction Linguistic Code Switching (LCS) is a common practice among multilingual speakers in which they switch between their common languages in written and spoken communication. In Spanish-English for example: “She told me that mi esposo looks like un buen hombre.” (“She told me that my husband looks like a good man”). In this work we care about detecting LCS points as they occur intra-sententially where words from more than one language are mixed in the same utterance. LCS is observed on all levels of linguistic representation, and especially pervasive in social media. LCS poses a significant challenge to NLP, hence detecting LCS points is a very important task for many downstream applications. In this paper we address this challenge using a generic simple language independent approach. We illustrate our approach on several language pairs utilizing publicly available data sets and comparing our performance against state-of-the-art sophisticated systems tailored to the problem of LCS point detection (LCSPD). Furthermore, we show the robustness of our approach on the most challenging problem of language variety code switching where the code switching is happening between a standard and dialect, namely we illustrate our performance on Modern Standard Arabic (MSA) mixed with Egyptian Dialectal Arabic data (EGY). 2 Related Work Several systems have recently addressed the problem of LCSPD in written text both within language varieties and across different language pairs. Relevant work on the problem of LCSPD among different language pairs can be summarized in the following works. 3ARRIB (Al-Badrashiny et al., 2014; Eskander et al., 2014) addresses the challenge of how to distin- guish between Arabic words written using Roman script (Arabizi) and actual English words in the same context/utterance. The assumption in this framework is that the script is Latin for all words. It trains a finite state transducer (FST) to learn the mapping between the Roman form …

Content truncated to 500 words