Modeling Code-Switch Languages Using Bilingual Parallel Corpus

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 860–870 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 860 Modeling Code-Switch Languages Using Bilingual Parallel Corpus Grandee Lee National University of Singapore grandee.lee@u.nus.edu Haizhou Li National University of Singapore Kriston AI Lab, China haizhou.li@nus.edu.sg Abstract Language modeling is the technique to esti- mate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across lan- guages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual se- quential dependency. The attention mecha- nism learns the bilingual context from a par- allel corpus. BALM achieves state-of-the- art performance on the SEAME code-switch database by reducing the perplexity of 20.5% over the best-reported result. We also apply BALM in bilingual lexicon induction, and lan- guage normalization tasks to validate the idea. 1 Introduction Monolingual language modeling has enabled many NLP tasks (Devlin et al., 2019; Dai et al., 2019; Radford et al., 2019). However, the bilingual lan- guage model was not well studied. The recent ad- vances in cross-lingual word embedding (CLWE) (Ruder et al., 2019), which projects word of differ- ent languages into a shared embedding space for cross-lingual representations (Devlin et al., 2019; Lample and Conneau, 2019), make possible some cross-lingual applications. Unfortunately, they are not optimized to model the sequential dependency for word prediction in a bilingual text. In this paper, we would like to propose a bilin- gual language model that can learn word embed- dings to represent the equivalent words between two languages, and more importantly, to model the sequential dependency for words across languages at the same time. For instance, the model should be able to predict the appropriate word to fill in the blank, given the bilingual context: 昨 晚 的 movie ( ). 1 The above sentence is an example of code- switching or code-mixing (henceforth, CS), where a bilingual speaker alternates words of two or more languages within a single sentence. The switches could happen at sentence boundaries or word boundaries and for some agglutinative lan- guages even within words. Code-switching is com- mon in both spoken and, to some extent, writ- ten communication in many multilingual societies, such as Southeast Asia. Hence, the study of code- switch in linguistics and bilingual language model- ing is becoming imperative, especially for NLP tasks such as code-switching automatic speech recognition (ASR) (Adel et al., 2013b; Li and Fung, 2013; Lee et al., 2019), cross-lingual language nor- malization. It is tempting to think that, given enough of code- switching text data, bilingual language modeling could be approached in the same way as that for monolingual data. The main challenge is the lack of such CS data. We note that CS mainly occurs …

Content truncated to 500 words