A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2267–2280 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2267 A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning Deepak Gupta, Asif Ekbal, Pushpak Bhattacharyya Indian Institute of Technology Patna, India {deepak.pcs16, asif, pb}@iitp.ac.in Abstract Code-mixing, the interleaving of two or more languages within a sentence or discourse is ubiquitous in multilingual societies. The lack of code-mixed training data is one of the ma- jor concerns for the development of end-to-end neural network-based models to be deployed for a variety of natural language processing (NLP) applications. A potential solution is to either manually create or crowd-source the code-mixed labelled data for the task at hand, but that requires much human efforts and of- ten not feasible because of the language spe- cific diversity in the code-mixed text. To cir- cumvent the data scarcity issue, we propose an effective deep learning approach for automati- cally generating the code-mixed text from En- glish to multiple languages without any paral- lel data. In order to train the neural network, we create synthetic code-mixed texts from the available parallel corpus by modelling various linguistic properties of code-mixing. Our code- mixed text generator is built upon the encoder- decoder framework, where the encoder is aug- mented with the linguistic and task-agnostic features obtained from the transformer based language model. We also transfer the knowl- edge from a neural machine translation (NMT) to warm-start the training of code-mixed gen- erator. Experimental results and in-depth anal- ysis show the effectiveness of our proposed code-mixed text generation on eight diverse language pairs. 1 Introduction Multilingual content is very prominent on social media handles, especially in the multilingual com- munities like the Indian ones. Code-mixing is a common expression of multilingualism in infor- mal text and speech, where there is a switch be- tween the two languages, frequently with one in the character set of the other language. This has been a mean of communication in a multi-cultural and multi-lingual society, and varies according to the culture, beliefs, and moral values of the respec- tive communities. Linguists have studied the phenomenon of code- mixing, put forward many linguistic hypotheses (Belazi et al., 1994; Pfaff, 1979; Poplack, 1978), and formulated various constraints (Sankoff and Poplack, 1981; Di Sciullo et al., 1986; Joshi, 1982) to define a general rule for code-mixing. How- ever, for all the scenarios of code-mixing, par- ticularly for the syntactically divergent languages (Berk-Seligson, 1986), these limitations cannot be postulated as a universal rule. In recent times, the pre-trained language model based architectures (Devlin et al., 2019; Radford et al., 2019) have become the state-of the-art models for language understanding and genera- tion. The underlying data to train such models comes from the huge amount of corpus, avail- able in the form of Wikipedia, book corpus etc. Although, these are readily available in various languages, there is a scarcity of such amount of data in code-mixed form which could be used …

Content truncated to 500 words