Universal Dependency Parsing for Hindi-English Code-Switching

Proceedings of NAACL-HLT 2018, pages 987–998 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Universal Dependency Parsing for Hindi-English Code-switching Irshad Ahmad Bhat LTRC, IIIT-H, Hyderabad, India irshad.bhat@iiit.ac.in Riyaz Ahmad Bhat Interaction Labs, Bangalore, India rbhat@interactions.com Manish Shrivastava LTRC, IIIT-H, Hyderabad, India m.shrivastava@iiit.ac.in Dipti Misra Sharma LTRC, IIIT-H, Hyderabad, India dipti@iiit.ac.in Abstract Code-switching is a phenomenon of mixing grammatical structures of two or more lan- guages under varied social constraints. The code-switching data differ so radically from the benchmark corpora used in NLP com- munity that the application of standard tech- nologies to these data degrades their perfor- mance sharply. Unlike standard corpora, these data often need to go through additional pro- cesses such as language identification, nor- malization and/or back-transliteration for their efficient processing. In this paper, we in- vestigate these indispensable processes and other problems associated with syntactic pars- ing of code-switching data and propose meth- ods to mitigate their effects. In particular, we study dependency parsing of code-switching data of Hindi and English multilingual speak- ers from Twitter. We present a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and propose a neural stacking model for parsing that effi- ciently leverages part-of-speech tag and syn- tactic tree annotations in the code-switching treebank and the preexisting Hindi and En- glish treebanks. We also present normaliza- tion and back-transliteration models with a decoding process tailored for code-switching data. Results show that our neural stacking parser is 1.5% LAS points better than the aug- mented parsing model and our decoding pro- cess improves results by 3.8% LAS points over the first-best normalization and/or back- transliteration. 1 Introduction Code-switching1 (henceforth CS) is the juxtaposi- tion, within the same speech utterance, of gram- matical units such as words, phrases, and clauses 1Code-mixing is another term in the linguistics literature used interchangeably with code-switching. Both terms are of- ten used to refer to the same or similar phenomenon of mixed language use. belonging to two or more different languages (Gumperz, 1982). The phenomenon is prevalent in multilingual societies where speakers share more than one language and is often prompted by mul- tiple social factors (Myers-Scotton, 1995). More- over, code-switching is mostly prominent in col- loquial language use in daily conversations, both online and offline. Most of the benchmark corpora used in NLP for training and evaluation are based on edited mono- lingual texts which strictly adhere to the norms of a language related, for example, to orthography, morphology, and syntax. Social media data in gen- eral and CS data, in particular, deviate from these norms implicitly set forth by the choice of corpora used in the community. This is the reason why the current technologies often perform miserably on social media data, be it monolingual or mixed lan- guage data (Solorio and Liu, 2008b; Vyas et al., 2014; C¸ etino˘glu et al., 2016; Gimpel et al., 2011; Owoputi et al., 2013; Kong et al., 2014). CS data offers additional challenges over the monolingual social media data as …

Content truncated to 500 words