Functions of Code-Switching in Tweets: An Annotation Framework and Some Initial Experiments

Functions of Code-Switching in Tweets: An Annotation Scheme and Some Initial Experiments Rafiya Begum, Kalika Bali, Monojit Choudhury, Koustav Rudra, Niloy Ganguly Microsoft Research Labs Indian Institute of Technology Bangalore, India Kharagpur, India {t-rafbeg, kalikab, monojitc}@microsoft.com krudra5@gmail.com, ganguly.niloy@googlemail.com Abstract Code-Switching (CS) between two languages is extremely common in communities with societal multilingualism where speakers switch between two or more languages when interacting with each other. CS has been extensively studied in spoken language by linguists for several decades but with the popularity of social-media and less formal Computer Mediated Communication, we now see a big rise in the use of CS in the text form. This poses interesting challenges and a need for computational processing of such code-switched data. As with any Computational Linguistic analysis and Natural Language Processing tools and applications, we need annotated data for understanding, processing, and generation of code-switched language. In this study, we focus on CS between English and Hindi Tweets extracted from the Twitter stream of Hindi-English bilinguals. We present an annotation scheme for annotating the pragmatic functions of CS in Hindi-English (Hi-En) code-switched tweets based on a linguistic analysis and some initial experiments. Keywords: Code-Switching, Corpus (Creation, Annotation, etc.), Multilinguality, Hindi-English, Social-Media Processing, Twitter, Pragmatic Functions 1. Introduction Code-Switching (CS) or switch between two or more languages in the context of a single conversation is a well- studied phenomenon in multilingual communities. The rise in social-media and other forms of Computer Mediated Communication (CMC) has seen CS, earlier associated more with spoken language, being used in the written form (Bali et al., 2014). The nature and extent of CS depends on a number of factors, including structural, pragmatic (functional), and socio-cultural aspects. Several studies (Labov, 1971; Joshi, 1985; Poplack, 1980) have indicated that CS is controlled by certain linguistic constraints at the structural level. At the functional level, it is generally considered as a conversational strategy to convey various distinct functions within a conversation (Barredo, 1997; Sanchez, 1983; Maschler, 1991; Blom and Gumprez, 1972). Other studies such as Annamalai (2001), Malhotra (1980), etc., investigate the social factors (e.g. age and socioeconomic status) effecting the nature of CS. Previous linguistic studies have looked at the structural and functional aspects of spoken and hence, small scale code-switched data. However, with the huge amount of text available on social-media there is now an opportunity to study different aspects of this phenomenon on a large scale. With the advent of speech-like conversational interaction on social-media, there has been a recent surge of interest in processing CS data. These studies are mostly in the areas of: (a) Language identification (Solorio et al., 2014), and (b) POS tagging (Solorio et al., 2008; Vyas et al., 2014). The computational processing of code-switched data is a challenging task from the perspective of linguistic understanding vis-à-vis discourse and conversational analysis, as well as computational modelling and applications to Machine Translation, Information Retrieval, and Natural Interfaces. For an in-depth understanding of why (pragmatic aspects) and how (structural aspects) people code-switch, we need data …

Content truncated to 500 words