Shallow Parsing Pipeline - Hindi-English Code-Mixed Social Media Text

Proceedings of NAACL-HLT 2016, pages 1340–1345, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Shallow Parsing Pipeline for Hindi-English Code-Mixed Social Media Text Arnav Sharma and Sakshi Gupta and Raveesh Motlani and Piyush Bansal and Manish Shrivastava and Radhika Mamidi and Dipti M. Sharma Kohli Center on Intelligent Systems (KCIS) International Institute of Information Technology, Hyderabad (IIIT Hyderabad) Gachibowli, Hyderabad, Telangana 500032 {arnav.s, sakshi.gupta, raveesh.motlani, piyush.bansal}@research.iiit.ac.in {m.shrivastava, radhika, dipti}@iiit.ac.in Abstract In this study, the problem of shallow pars- ing of Hindi-English code-mixed social me- dia text (CSMT) has been addressed. We have annotated the data, developed a language identifier, a normalizer, a part-of-speech tag- ger and a shallow parser. To the best of our knowledge, we are the first to attempt shallow parsing on CSMT. The pipeline developed has been made available to the research commu- nity with the goal of enabling better text anal- ysis of Hindi English CSMT. The pipeline is accessible at 1. 1 Introduction Multilingual speakers tend to exhibit code-mixing and code-switching in their use of language on so- cial media platforms. Code-Mixing is the embed- ding of linguistic units such as phrases, words or morphemes of one language into an utterance of an- other language whereas code-switching refers to the co-occurrence of speech extracts belonging to two different grammatical systems (Gumperz., 1982). Here we use code-mixing to refer to both the sce- narios. Hindi-English bilingual speakers produce huge amounts of CSMT. Vyas et al. (2014) noted that the complexity in analyzing CSMT stems from non- adherence to a formal grammar, spelling variations, lack of annotated data, inherent conversational na- ture of the text and of course, code-mixing. There- fore, there is a need to create datasets and Natural 1http://bit.ly/csmt-parser-api Language Processing (NLP) tools for CSMT as tra- ditional tools are ill-equipped for it. Taking a step in this direction, we describe the shallow parsing pipeline built during this study. 2 Background Bali et al. (2014) gathered data from Facebook generated by English-Hindi bilingual users which on analysis, showed a significant amount of code- mixing. Barman et al. (2014) investigated lan- guage identification at word level on Bengali-Hindi- English CSMT. They annotated a corpus with more than 180,000 tokens and achieved an accuracy of 95.76% using statistical models with monolingual dictionaries. Solorio and Liu (2008) experimented with POS tagging for English-Spanish Code-Switched dis- course by using pre-existing taggers for both lan- guages and achieved an accuracy of 93.48%. How- ever, the data used was manually transcribed and thus lacked the problems added by CSMT. Vyas et al. (2014) formalized the problem, reported chal- lenges in processing Hindi-English CSMT and per- formed initial experiments on POS tagging. Their POS tagger accuracy fell by 14% to 65% without using gold language labels and normalization. Thus, language identification and normalization are criti- cal for POS tagging (Vyas et al., 2014), which in turn is critical further down the pipeline for shallow parsing as evident in Table 5. Jamatia et al. (2015) also built a POS tag- …

Content truncated to 500 words