Comparing the Level of Code-Switching in Corpora

Comparing the Level of Code-Switching in Corpora Björn Gambäck∗, Amitava Das† ∗Department of Computer and Information Science Norwegian University of Science and Technology NO–7491 Trondheim, Norway gamback@idi.ntnu.no †Indian Institute of Information Technology Sri City, Satyavedu Mandal, Chittoor District Andhra Pradesh – 517588, India amitava.das@iiits.in Abstract Social media texts are often fairly informal and conversational, and when produced by bilinguals tend to be written in several different languages simultaneously, in the same way as conversational speech. The recent availability of large social media corpora has thus also made large-scale code-switched resources available for research. The paper addresses the issues of evaluation and comparison these new corpora entail, by defining an objective measure of corpus level complexity of code-switched texts. It is also shown how this formal measure can be used in practice, by applying it to several code-switched corpora. Keywords: Code-switching, Evaluation, Corpora, Social Media Text 1. Introduction When two individuals who are bi- or multi-lingual in an overlapping set of languages communicate, they tend to switch seemlessly and effortlessly between the languages (codes) they share. When this code alternation occurs at or above the utterance level, the phenomenon is referred to as code-switching; when the alternation is utterance- internal, the term ‘code-mixing’ is common, even though ‘code-switching’ is frequently used in those cases as well. Code-mixing in itself is often an effect of what recently (particularly within language teaching) has started to be called ‘translanguaging’, that is, when truly bi-lingual in- dividuals are creating new meanings based on their full and double language repertoire (Lewis et al., 2012). Code-switching is most prominent in spoken language con- versations and has thus traditionally mainly been studied by psycho- and sociolinguists (Auer, 1999; Muysken, 2000; Gafaranga and Torras, 2002; Bullock et al., 2014) and by speech researchers (Lyu et al., 2015), while the lack of large-scale textual corpora has made code-switching less attractive as a subject of study in computational or corpora linguistics. However, this started changing in 2003 with the advent of social media, where large amounts of texts are written that are more informal and more conversational in nature, and hence when produced by bilinguals tend to contain more code-switching (Paolillo, 1996). This new availability of large-scale code-switched re- sources in turn raises questions of evaluation: how do we compare the results of applying language processing tools to one code-switched corpus to those on another? Or more specifically: how can we compare the level of code- switching in corpora? And for corpora containing a mix of a specific set of languages or across corpora from differ- ent sets of languages? These are the issues that the present paper aims to address. That two texts come from social media does not in itself imply that they belong to one, delimited textual domain. Rather, there is a wide spectrum of different types of texts that are transmitted through social media, and the level of formality of the language in addition depends more on the style of the writer than on the actual media …

Content truncated to 500 words