Minority Positive Sampling for Switching Points - an Anecdote for the Code-Mixing Language Modeling

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6228–6236 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6228 Minority Positive Sampling for Switching Points - an Anecdote for the Code-Mixing Language Modeling Arindam Chatterjee¹, Bodla Vineeth Guptha², Parul Chopra³, Amitava Das Wipro AI Labs Bangalore, India {arindam.chatterjee4, bodla.guptha, parul.chopra1, amitava.das2}@wipro.com Abstract Code-Mixing (CM) or language mixing is a social norm in multilingual societies. CM is quite prevalent in social media conversations in multilingual regions like - India, Europe, Canada and Mexico. In this paper, we explore the problem of Language Modeling (LM) for code-mixed Hinglish (Hindi-English language pair) text. In recent times, there have been several success stories with neural language modeling like Generative Pre-trained Transformer (GPT) (Radford et al., 2019), Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) etc.. Hence, neural language models have become the new holy grail of modern NLP, although LM for CM is an unexplored area altogether. To better understand the problem of LM for CM, we initially experimented with several statistical language modeling techniques and consequently experimented with contemporary neural language models. Analysis shows that switching points (junctions in the text where the language switches) are the main challenge for the CM language model and the reason for the performance drop, as compared to monolingual LMs. To handle this impediment, in this paper we introduce the concept of minority positive sampling, to selectively induce more samples, to achieve better performance. The neural language models for CM demand a huge corpus, still they exhibit improvement in performance, after the samples are induced. Finally, we report a perplexity of 139 for Hinglish LM for CM using statistical bi-directional technique. Keywords: code-mix, language modelling, Hinglish, switching point 1. Introduction Mixing languages, also known as code-mixing, is a norm in multilingual societies. Multilingual people, who are non-native English speakers, tend to code-mix using English-based phonetic typing and insertion of anglicisms in their native language. In addition to mix- ing languages at the sentence level, it is also fairly com- mon to find code-mixing behavior at the word level. This linguistic phenomenon poses a great challenge to conventional NLP systems. The following phrase is an example of code-mixing in Hinglish i.e., mixed between Hindi and English. In the example only one English word enjoy has been used, but more noticeably for the Hindi words - instead of using the native Devanagari script, English phonetic typing is a popular choice. AyeHI aurHI enjoyEN kareHI Eng. Trans.: come and enjoy Naturally, code-mixing is more common in geograph- ical regions with a high percentage of bi or multilin- gual speakers, such as in Texas and California in the US, Hong Kong and Macao in China, many European and African countries, and the countries in South-East Asia. Multi-linguality and code-mixing are also very common in India. Code-mixing has received a significant attention very recently, mainly due to the availability of large-scale code-mixed data in blogs, micro-blogs (e.g., Twitter), WhatsApp, and chats …

Content truncated to 500 words