Estimating Code-Switching on Twitter with a Novel Generalized Word-Level Language Detection Technique

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1971–1982 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1180 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1971–1982 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1180 Estimating Code-Switching on Twitter with a Novel Generalized Word-Level Language Detection Technique Shruti Rijhwani∗ Language Technologies Institute Carnegie Mellon University srijhwan@cs.cmu.edu Royal Sequiera∗ University of Waterloo Waterloo, Canada rdsequie@uwaterloo.ca Monojit Choudhury Kalika Bali Microsoft ResearchChandra Sekhar Maddila Bangalore, India {monojitc,kalikab,chmaddil}@microsoft.com Abstract Word-level language detection is neces- sary for analyzing code-switched text, where multiple languages could be mixed within a sentence. Existing models are restricted to code-switching between two specific languages and fail in real-world scenarios as text input rarely has a priori information on the languages used. We present a novel unsupervised word-level language detection technique for code- switched text for an arbitrarily large num- ber of languages, which does not require any manually annotated training data. Our experiments with tweets in seven lan- guages show a 74% relative error reduc- tion in word-level labeling with respect to competitive baselines. We then use this system to conduct a large-scale quanti- tative analysis of code-switching patterns on Twitter, both global as well as region- specific, with 58M tweets. 1 Introduction In stable multilingual societies, communication often features fluid alteration between two or more languages – a phenomenon known as code-switching1 (Gumperz, 1982; Myers-Scotton, 1993). It has been studied extensively in linguis- tics, primarily as a speech phenomenon (Poplack, 1980; Gumperz, 1982; Myers-Scotton, 1993; Mil- roy and Muysken, 1995; Auer, 2013). How- ever, the growing popularity of computer mediated ∗* This work was done when the authors were affiliated with Microsoft Research. 1This paper uses the terms ‘code-switching’ and ‘code- mixing’ interchangeably. communication, particularly social media, has re- sulted in language data in the text form which exhibits code-switching, among other speech- like characteristics (Crystal, 2001; Herring, 2003; Danet and Herring, 2007; Cardenas-Claros and Isharyanti, 2009). With the large amount of online content generated by multilingual users around the globe, it becomes necessary to design techniques to analyze mixed language, which can help not only in developing end-user applications, but also in conducting fundamental sociolinguistic studies. Language detection (LD) is a prerequisite to several NLP techniques. Most state-of-the-art LD systems detect a single language for an entire doc- ument or sentence. Such methods often fail to detect code-switching, which can occur within a sentence. In recent times, there has been some effort to build word-level LD for code-switching between a specific pair of languages (Nguyen and Dogru¨oz, 2013; Elfardy et al., 2013; Solorio et al., 2014; Barman et al., 2014). However, usually user-generated text (e.g., on social media) has no prior information of the languages being used. Further, as several previous social-media based studies on multilingualism have pointed out (Kim et al., 2014; Manley, 2012), lack of general word- level LD has been a bottleneck in …

Content truncated to 500 words