Word Level Language Identification in Online Multilingual Communication

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 857–862, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Word Level Language Identification in Online Multilingual Communication Dong Nguyen1 A. Seza Do˘gru¨oz23 (1) Human Media Interaction, University of Twente, Enschede, The Netherlands (2) Tilburg School of Humanities, Tilburg University, Tilburg, The Netherlands (3) Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA dong.p.ng@gmail.com, a.s.dogruoz@gmail.com Abstract Multilingual speakers switch between lan- guages in online and spoken communication. Analyses of large scale multilingual data re- quire automatic language identification at the word level. For our experiments with mul- tilingual online discussions, we first tag the language of individual words using language models and dictionaries. Secondly, we incor- porate context to improve the performance. We achieve an accuracy of 98%. Besides word level accuracy, we use two new metrics to evaluate this task. 1 Introduction There are more multilingual speakers in the world than monolingual speakers (Auer and Wei, 2007). Multilingual speakers switch across languages in daily communication (Auer, 1999). With the in- creasing use of social media, multilingual speakers also communicate with each other in online environ- ments (Paolillo, 2011). Data from such resources can be used to study code switching patterns and lan- guage preferences in online multilingual conversa- tions. Although most studies on multilingual online communication rely on manual identification of lan- guages in relatively small datasets (Danet and Her- ring, 2007; Androutsopoulos, 2007), there is a grow- ing demand for automatic language identification in larger datasets. Such a system would also be useful for selecting the right parsers to process multilingual documents and to build language resources for mi- nority languages (King and Abney, 2013). In this paper, we identify Dutch (NL) en Turkish (TR) at the word level in a large online forum for Turkish-Dutch speakers living in the Netherlands. The users in the forum frequently switch languages within posts, for example: <TR> Sariyi ver </TR> <NL> Wel mooi doelpunt </NL> So far, language identification has mostly been mod- eled as a document classification problem. Most ap- proaches rely on character or byte n-grams, by com- paring n-gram profiles (Cavnar and Trenkle, 1994), or using various machine learning classifiers. While McNamee (2005) argues that language identification is a solved problem, classification on a more fine- grained level (instead of document level) remains a challenge (Hughes et al., 2006). Furthermore, lan- guage identification is more difficult for short texts (Baldwin and Lui, 2010; Vatanen et al., 2010), such as queries and tweets (Bergsma et al., 2012; Carter et al., 2012; Ceylan and Kim, 2009). Tagging in- dividual words (without context) has been done us- ing dictionaries, affix statistics and classifiers us- ing character n-grams (Hammarstr¨om, 2007; Got- tron and Lipka, 2010). Although Yamaguchi and Tanaka-Ishii (2012) segmented text by language, their data was artificially created by randomly sam- pling and concatenating text segments (40-160 char- acters) from monolingual texts. Therefore, the lan- guage switches do not reflect realistic switches as they occur in natural …

Content truncated to 500 words