POS Tagging of English-Hindi Code-Mixed Social Media Content

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 974–979, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics POS Tagging of English-Hindi Code-Mixed Social Media Content Yogarshi Vyas∗ University of Maryland yogarshi@cs.umd.edu Spandana Gella∗ Xerox Research Centre Europe spandanagella@gmail.com Jatin Sharma Kalika Bali Monojit Choudhury Microsoft Research India {jatin.sharma,kalikab,monojitc}@microsoft.com Abstract Code-mixing is frequently observed in user generated content on social media, especially from multilingual users. The linguistic complexity of such content is compounded by presence of spelling vari- ations, transliteration and non-adherance to formal grammar. We describe our initial efforts to create a multi-level an- notated corpus of Hindi-English code- mixed text collated from Facebook fo- rums, and explore language identifica- tion, back-transliteration, normalization and POS tagging of this data. Our re- sults show that language identification and transliteration for Hindi are two major challenges that impact POS tagging accu- racy. 1 Introduction Code-Switching and Code-Mixing are typical and well-studied phenomena of multilingual so- cieties (Gumperz, 1964; Auer, 1984; Myers- Scotton, 1993; Danet and Herring, 2007; Cardenas-Claros and Isharyanti, 2009). Lin- guists differentiate between the two, where Code-Switching is juxtaposition within the same speech exchange of passages of speech be- longing to two different grammatical systems or sub-systems (Gumperz, 1982), and Code- Mixing (CM) refers to the embedding of linguis- tic units such as phrases, words and morphemes of one language into an utterance of another lan- guage (Myers-Scotton, 1993). The first exam- ple in Fig. 1 features CM where English words are embedded in a Hindi sentence, whereas the second example shows codeswitching. Here, we will use CM to imply both. Work on computa- ∗This work was done during authors’ internship at Mi- crosoft Research India. tional models of CM have been few and far be- tween (Solorio and Liu, 2008a; Solorio and Liu, 2008b; Nguyen and Dogruoz, 2013), primarily due to the paucity of CM data in conventional text-corpora which makes data-intensive methods hard to apply. Solorio and Liu (2008a) in their work on English-Spanish CM use models built on smaller datasets to predict valid switching points to synthetically generate data from monolingual corpora, and in another work (2008b) describe parts-of-speech (POS) tagging of CM text. CM though typically observed in spoken lan- guage is now increasingly more common in text, thanks to the proliferation of the Computer Me- diated Communication channels, especially so- cial media like Twitter and Facebook (Crys- tal, 2001; Herring, 2003; Danet and Herring, 2007; Cardenas-Claros and Isharyanti, 2009). Social media content is tremendously important for studying trends, reviews, events, human- behaviour as well as linguistic analysis, and there- fore in recent times has spurred a lot of interest in automatic processing of such data. Neverthe- less, CM on social media has not been studied from a computational aspect. Moreover, social media content presents additional challenges due to contractions, non-standard spellings and non- grammatical constructions. Furthermore, for lan- guages written in scripts other than Roman, like Hindi, Bangla, Japanese, Chinese and Arabic, Ro- man transliterations are typically …

Content truncated to 500 words