All that is English may be Hindi: Enhancing language identification through automatic ranking of the likeliness of word borrowing in social media

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2264–2274 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media 1Jasabanta Patro, 2Bidisha Samanta, 2Saurabh Singh, 2Abhipsa Basu,2Prithwish Mukherjee, 3Monojit Choudhury, 4Animesh Mukherjee 1,2,4 Indian Institute of Technology Kharagpur, India – 721302 6Microsoft Research India, Bangalore – 560001 1jasabantapatro@iitkgp.ac.in, 3monojitc@microsoft.com, 4animeshm@cse.iitkgp.ernet.in Abstract In this paper, we present a set of compu- tational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman’s correlation values, our meth- ods perform more than two times better (∼ 0.62) in predicting the borrowing like- liness compared to the best performing baseline (∼ 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88% of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus in- dicating a huge scope for improvement of automatic language identification systems. 1 Introduction In social media communication, multilingual peo- ple often switch between languages, a phe- nomenon known as code-switching or code- mixing (Auer, 1984). This makes language iden- tification and tagging, which is perhaps a pre- requisite for almost all other language processing tasks that follow, a challenging problem (Barman et al., 2014). In code-mixing people are subcon- sciously aware of the foreign origin of the code- mixed word or the phrase. A related but linguisti- cally and cognitively distinct phenomenon is lex- ical borrowing (or simply, borrowing), where a word or phrase from a foreign language say L2 is used as a part of the vocabulary of native lan- guage say L1. For instance, in Dutch, the En- glish word “sale” is now used more frequently than the Dutch equivalent “uitverkoop”. Some English words like “shop” are even inflected in Dutch as “shoppen” and heavily used. While it is difficult in general to ascertain whether a foreign word or phrase used in an utterance is borrowed or just an instance of code-mixing (Bali et al., 2014), one tell tale sign is that only proficient mul- tilinguals can code-mix, while even monolingual speakers can use borrowed words because, by def- inition, these are part of the vocabulary of a lan- guage. In other words, just because an English speaker understands and uses the word “tortilla” does not imply that she can speak or understand Spanish. A borrowed word from L2 initially ap- pears frequently in speech, then gradually in print media like newspaper and finally it loses its ori- gin’s identity and is used in L1 resulting in an in- clusion in the dictionary of L1 (Myers-Scotton, 2002; Thomason, 2003). Borrowed words often take several years before they formally become part of L1 dictionary. This motivates our research question “is early-stage automatic identification of likely to be borrowed words possible?”. …

Content truncated to 500 words