Word-level Language Identification in Bi-lingual Code-switched Texts

Word-level Language Identification in Bi-lingual Code-switched Texts Abstract Code-switching is the practice of moving back and forth between two languages in spoken or written form of communication. In this paper, we address the problem of word-level language identification of code-switched sentences. Here, we primarily consider Hindi-English (Hinglish) code-switching, which is a popular phenomenon among urban Indian youth, though the approach is generic enough to be extended to other language pairs. Identifying word-level languages in code-switched texts is associated with two major challenges. Firstly, people often use non-standard English transliterated forms of Hindi words. Secondly, the transliterated Hindi words are often confused with English words having the same spelling. Most existing works tackle the problem of language identification using n-grams of characters. We propose some techniques to learn sequence of character(s) frequently substituted for character(s) in standard transliterated forms. We illustrate the superior performance of these techniques in identifying Hindi words corresponding to the given transliterated forms. We adopt a novel experimental model which considers the language and part-of-speech of adjoining words for word-level language identification. Our test results show that the proposed model significantly increases the accuracy over existing approaches. We achieved F1-score of 98.0% for recognizing Hindi words and 94.8% for recognizing English words. 1 Introduction Code-switching is a popular linguistic phenomenon where the speaker alternates between two or more languages even within the same sentence. In countries like India, where there are more than 20 widely used languages, code- switching is an even more pronounced feature, mostly among urban population (Thakur et al., 2007). Hindi and English are two popular ones among these languages, with millions of people communicating through them in pure forms or using a mixture of words from both the languages (code-switched), popularly known as ‘Hinglish’. Many multi-national brands use Hinglish taglines for promoting their products in India. For example, “Khushiyon ki home delivery”1 is the tag line for Domino’s Pizza TM India. Hinglish is also used for casual communication among friends, for example, “Main temple ke pass hoon” meaning “I am near the temple”. There are plenty of research works focusing on analyzing texts used in popular forums, like online social groups, for applications like opinion mining, sentiment analysis, etc. However, machine analysis of Hinglish or any other code-switched text poses the following challenges. • Inconsistent spelling usage: Despite the availability of the standards for transliteration (e.g., ITRANS 2 ) of Devanagari script to Roman script (the Hindi language is based on Devanagari script while the English language is based on Roman script), people tend to use many inconsistent spellings for the same word. For example, the most common English transliteration for the Hindi word मैं is mai, as observed from our data set. But people often use mein or main as alternatives. • Ambiguous word usage: The transliterated word main, for the Hindi 1 http://www.dominos.co.in/blog/tag/khushiyon-ki-home- delivery/ 2 ITRANS: http://www.aczoom.com/itrans/ Harsh Jhamtani Suleep Kumar Bhogi Vaskar Raychoudhury Adobe systems Samsung R&D institute Dept. of Comp. Sc. & Engg. Bangalore, India Bangalore, India IIT Roorkee, …

Content truncated to 500 words