Word Level Language Identification in English Telugu Code Mixed Data

Word Level Language Identification in English Telugu Code Mixed Data Sunil Gundapu Language Technologies Research Centre KCIS, IIIT Hyderabad Telangana, India gundapusunil@gmail.com Radhika Mamidi Language Technologies Research Centre KCIS, IIIT Hyderabad Telangana, India radhika.mamidi@iiit.ac.in Abstract In a multilingual or sociolingual configura- tion Intra-sentential Code Switching (ICS) or Code Mixing (CM) is frequently observed nowadays. In the world most of the people know more than one language. The CM us- age is especially apparent in social media plat- forms. Moreover, ICS is particularly signifi- cant in the context of technology, health and law where conveying the upcoming develop- ments are difficult in ones native language. In applications like dialog systems, machine translation, semantic parsing, shallow pars- ing, etc. CM and Code Switching pose seri- ous challenges. To do any further advance- ment in code-mixed data, the necessary step is Language Identification. So, in this pa- per we present a study of various models - Nave Bayes Classifier, Random Forest Clas- sifier, Conditional Random Field (CRF) and Hidden Markov Model (HMM) for Language Identification in English - Telugu Code Mixed Data. Considering the paucity of resources in code mixed languages, we proposed CRF model and HMM model for word level lan- guage identification. Our best performing sys- tem is CRF-based with an f1-score of 0.91. 1 Introduction Code switching is characterized as the use of two or more languages, diversity and verbal style by a speaker within a statement, pronouncement or discourse, or between different speakers or situa- tions. This code switching can be classified as inter- sentential (the language switching is done at sen- tence boundaries) and intra-sentential (the alterna- tion in a single discourse between two languages, where the switching occurs within a sentence). Code mixing is inconsistently elucidated in disparate sub- fields of linguistics and frequent examination of phrases, words, inflectional, derivational morphol- ogy and syntax use of a term as an equivalent to Code Mixing. Code mixing is defined as the embedding of lin- guistic units of one language into utterance of an- other language. CM is not only used in commonly used spoken form of multilingual setting, but also used in social media websites in the form of com- ments and replies, posts and especially in chat con- versations. Most of the chat conversations are in a formal or semi-formal setting and CM is often used. It commonly takes place in scenarios where a stan- dard formal education is received in a language dif- ferent than the persons native language or mother tongue. Most of the case studies state that CM is very pop- ular in the world in the present day, especially in countries like India, with more than 20 official lan- guages. One such language, Telugu is a Dravidian language used by a total of 7.19% people of Telan- gana and Andra Pradesh states in India. Because of influence of English, most people use a combination of Telugu and English in conversations. There is lot of research work being carried out in code mixed data …

Content truncated to 500 words