Part-of-Speech Tagging for Code-Mixed English-Hindi Twitter and Facebook Chat Messages

Proceedings of Recent Advances in Natural Language Processing, pages 239–248, Hissar, Bulgaria, Sep 7–9 2015. Part-of-Speech Tagging for Code-Mixed English-Hindi Twitter and Facebook Chat Messages Anupam Jamatia National Institute of Technology Agartala, Tripura, India anupamjamatia@gmail.com Björn Gambäck Norwegian University of Science and Technology Trondheim, Norway gamback@idi.ntnu.no Amitava Das Indian Institute of Information Technology Sri City, Andhra Pradesh, India amitava.das@iiits.in Abstract The paper reports work on collecting and annotating code-mixed English-Hindi so- cial media text (Twitter and Facebook messages), and experiments on automatic tagging of these corpora, using both a coarse-grained and a fine-grained part-of- speech tag set. We compare the perfor- mance of a combination of language spe- cific taggers to that of applying four ma- chine learning algorithms to the task (Con- ditional Random Fields, Sequential Mini- mal Optimization, Naïve Bayes and Ran- dom Forests), using a range of different features based on word context and word- internal information. 1 Introduction Code-mixing occurs when a person changes lan- guage (alternates or switches code) below clause level, so internally inside a sentence or an utter- ance. This phenomenon is more abundant in more informal settings — such as in conversational spo- ken language and in social media text — and of course also more common in areas of the world where people are naturally bi- or multilingual, that is, in regions where languages change over short geospatial distances and people generally have at least a basic knowledge of the neighbouring lan- guages. In particular, India is home to several hun- dred languages, with language diversity and di- alectal changes instigating frequent code-mixing. We will here look at the tasks of collecting and annotating code-mixed English-Hindi social me- dia text, and on automatic part-of-speech (POS) tagging of these code-mixed texts. In contrast, most research on part-of-speech tagging has so far concentrated on more formal language forms, and in particular either on completely monolingual text or on text where code alternation occurs above the clause level. Most research on social media text has, on the other hand, concentrated on En- glish tweets, whereas the majority of these texts now are written in other media and in other lan- guages — or in mixes of languages. Today, code-switching is generally recognised as a natural part of bi- and multilingual language use, even though it historically often was consid- ered a sub-standard use of language. Conversa- tional spoken language code-switching has been a common research theme in psycho- and socio- linguists for half a century, and the first work on applying language processing methods to code- switched text was carried out in the early 1980s (Joshi, 1982), while code-switching in social me- dia text started to be studied in the late 1990s (Pao- lillo, 1996). Still, code alternation in conventional texts is not so prevalent as to spur much interest by the computational linguistic research community, and it was only recently that it became a research topic in its own right, with a code-switching work- shop at EMNLP 2014 (Solorio et al., 2014), and …

Content truncated to 500 words