MHE: Code-Mixed Corpora for Similar Language Identification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3425–3433 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3425 MHE: Code-Mixed Corpora for Similar Language Identification Priya Rani, John P. McCrae, Theodorus Fransen Data Science Institute, National University of Ireland Galway Galway, Ireland {prani1, john.mccrae, theodorus.fransen}@nuigalway.ie Abstract This paper introduces a new Magahi-Hindi-English (MHE) code-mixed dataset for similar language identification, where Magahi is a less-resourced minority language. This corpus provides language identification at two levels: word and sentence. This dataset is the first Magahi-Hindi-English code-mixed dataset for similar language identification task. Furthermore, we will discuss the complexity of the dataset and provide a few baselines for the language identification task. Keywords: Magahi, Code-Mixing, Similar languages, Language Identification 1. Introduction Language identification has been called a “solved task” (McNamee, 2005) and while this may be the case for many languages, for closely-related languages this is still a challenging task (Goswami et al., 2020). In particular, for many closely-related languages there is a large degree of code-mixing in the typical usage of these languages and as such language identification changes from a sentence- or document-level task into a word-level task. In particular, we focus on the case of Magahi-Hindi-English code-mixed texts, where speak- ers of Magahi frequently mix their language with the most widely-spoken language in the country, Hindi, or with the international language of English. Magahi is not an official language of India and is often consid- ered as a Hindi dialect; however, the massive growth in and increasing access to social media means that the language is developing a new identity online. There is significant overlap in the vocabulary between Magahi and Hindi, which further adds to the challenge of this task: words may simply be of both languages with no distinction made, or words from one of the language may be used with the morphology and grammar of the other. As such this creates a challenging language iden- tification task for which there exist no corpora which can be used to develop systems. For less-resourced mi- nority languages such as Magahi, developing corpora is therefore a challenging task, and the availability of an- notators as well as the linguistic challenges of working with closely-related languages in code-mixed scenarios add substantial obstacles to corpus development. In this work, we present the first code-mixed corpus for Magahi-Hindi-English based on text collected from social media platforms. The text in our corpus is code-mixed text as is common in user-generated con- tent (Zhang et al., 2018). Code-switching occurs when conversants use both languages together to the extent that they change from one language to the other in the course of a single utterance (Wardhaugh and Fuller, 1986). This linguistic phenomenon is very prevalent in informal communication in a multilingual society where speakers switch between two or more languages and this is also a well-known practice among social me- dia users. The corpus we developed shows a substan- tially higher degree of code-mixing compared to …

Content truncated to 500 words