Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 271–280 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 271 Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {giwinata,amadotto,cwuak}@connect.ust.hk, pascale@ece.ust.hk Abstract Training code-switched language models is difficult due to lack of data and complexity in the grammatical structure. Linguistic con- straint theories have been used for decades to generate artificial code-switching sentences to cope with this issue. However, this require ex- ternal word alignments or constituency parsers that create erroneous results on distant lan- guages. We propose a sequence-to-sequence model using a copy mechanism to gener- ate code-switching data by leveraging paral- lel monolingual translations from a limited source of code-switching data. The model learns how to combine words from parallel sentences and identifies when to switch one language to the other. Moreover, it captures code-switching constraints by attending and aligning the words in inputs, without requiring any external knowledge. Based on experimen- tal results, the language model trained with the generated sentences achieves state-of-the- art performance and improves end-to-end au- tomatic speech recognition. 1 Introduction Code-switching is a common linguistic phe- nomenon in multilingual communities, in which a person begins speaking or writing in one lan- guage and then switches to another in the same sentence.1 It is motivated in response to social fac- tors as a way of communicating in a multicultural society. In its practice, code-switching varies due to the traditions, beliefs, and normative values in the respective communities. Linguists have stud- ied the code-switching phenomenon and proposed a number of linguistic theories (Poplack, 1978; Pfaff, 1979; Poplack, 1980; Belazi et al., 1994). Code-switching is not produced indiscriminately, 1Code-switching refers to mixing of languages following the definitions in Poplack (1980). We use “intra-sentential code-switching" interchangeably with “code-mixing". but follows syntactic constraints. Many linguists have formulated various constraints to define a general rule for code-switching (Poplack, 1978, 1980; Belazi et al., 1994). However, these con- straints cannot be postulated as a universal rule for all code-switching scenarios, especially for languages that are syntactically divergent (Berk- Seligson, 1986), such as English and Mandarin since they have word alignments with an inverted order. Building a language model (LM) and an auto- matic speech recognition (ASR) system that can handle intra-sentential code-switching is known to be a difficult research challenge. The main rea- son lies in the unpredictability of code-switching points in an utterance and data scarcity. Creating a large-scale code-switching dataset is also very expensive. Therefore, code-switching data gener- ation methods to augment existing datasets are a useful workaround. Existing methods that apply equivalence con- straint theory to generate code-switching sen- tences (Li and Fung, 2012; Pratapa et al., 2018) may suffer performance issues as they receive er- roneous results from the word aligner and the part- of-speech (POS) …

Content truncated to 500 words