Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text

Proceedings of Recent Advances in Natural Language Processing, pages 744–752 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_085 744 Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text Kusampudi Siva Subrahamanyam Varma, Anudeep Chaluvadi, Radhika Mamidi Language Technologies Research Centre, IIIT Hyderabad, Telangana, India siva.subrahamanyam@research.iiit.ac.in anudeep.chaluvadi@students.iiit.ac.in radhika.mamidi@iiit.ac.in Abstract Code-Mixing (CM) is a common phe- nomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. Language Identification (LID) of the CM data will help solve NLP tasks such as Spell Checking, Named Entity Recogni- tion, Parts-Of-Speech tagging, and Seman- tic Parsing. In the current era of ma- chine learning, a common problem to the above-mentioned tasks is the availability of Learning data to train models. In this paper, we introduce two Telugu-English CM manually annotated datasets (Twitter dataset and Blog dataset). The Twitter dataset contains more romanization vari- ability and misspelled words than the blog dataset. We compare across various clas- sification models and perform extensive bench-marking using both Classical and Deep Learning Models for LID compared to existing models. We propose two archi- tectures for language classification (Telugu and English) in CM data: (1) Word Level Classification (2) Sentence Level word-by- word Classification and compare these ap- proaches presenting two strong baselines for LID on these datasets. 1 Introduction Language is one of the significant aspects which makes humans different from other species. It is not a fixed entity, and it has evolved with time and will continue to do so. As a part of such an evolutionary process, we are at the stage of Code-Mixing where people communicate by mixing linguistic units such as phrases, words, and morphemes of one lan- guage embedded within an utterance of an- other language (Sankoff and Poplack, 1981), (Poplack, 1980). India has 22 officially recognized languages1 and many dialects. For a land with such linguistic diversity, bilingualism and multilin- gualism is a prevalent trait. Telugu belongs to the family of Dravidian languages. It is pri- marily spoken in Southern India and is also the third most spoken language in India. Telugu is the mother language of a large population na- tive of Andhra Pradesh and Telangana states. English is a primary mode of teaching for most of the population across the globe. With such an influence of a language, people tend to use a mix of both English and their native language in an informal conversation (both speech and text). CM is classified into two types: Intra- Sentential and Inter-Sentential Code-Mixing (Zirker, 2007). Intra-Sentential Code Mixing refers to the use of multiple languages in a sin- gle sentence. Inter-Sentential Code-Mixing is when the language switching is done at the end of the sentence. In this paper, we will focus on Intra-Sentential Code Mixing. In a multilingual setting, most of the con- versations that happen informally are CM. CM is also used extensively in Social Media platforms, blogs, and forums as posts, chats, and comments. The processing of CM text poses an …

Content truncated to 500 words