Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4175–4185, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4175 Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training Hila Gonen1 and Yoav Goldberg1,2 1Department of Computer Science, Bar-Ilan University 2Allen Institute for Artificial Intelligence {hilagnn,yoav.goldberg}@gmail.com Abstract We focus on the problem of language model- ing for code-switched language, in the con- text of automatic speech recognition (ASR). Language modeling for code-switched lan- guage is challenging for (at least) three rea- sons: (1) lack of available large-scale code- switched data for training; (2) lack of a repli- cable evaluation setup that is ASR directed yet isolates language modeling performance from the other intricacies of the ASR system; and (3) the reliance on generative modeling. We tackle these three issues: we propose an ASR- motivated evaluation setup which is decoupled from an ASR system and the choice of vo- cabulary, and provide an evaluation dataset for English-Spanish code-switching. This setup lends itself to a discriminative training ap- proach, which we demonstrate to work better than generative language modeling. Finally, we explore a variety of training protocols and verify the effectiveness of training with large amounts of monolingual data followed by fine- tuning with small amounts of code-switched data, for both the generative and discrimina- tive cases. 1 Introduction This work deals with neural language modeling of code-switched language, motivated by an ap- plication to speech recognition. Code-switching (CS) is a linguistic phenomenon defined as “the alternation of two languages within a single dis- course, sentence or constituent.” (Poplack, 1980). Since CS is widely used in spoken platforms, deal- ing with code-switched language becomes an im- portant challenge for automatic speech recognition (ASR) systems. To get a feeling how an ASR system trained on monolingual data performs on code-switched language, we fed the IBM English and Spanish systems1 with audio files of code- switched conversations from the Bangor Miami Corpus (see Section 7). The results (examples available in Table 1) exhibit two failure modes: (1) Words and sentences in the opposite language are not recognized correctly; and (2) Code-switch points also hurt recognition of words from the main language. This demonstrates the need for designated speech recognition systems for CS. A crucial component in such a CS ASR system is a strong CS language model, which is used to rank the bilingual candidates produced by the ASR sys- tem for a given acoustic signal. Language models are traditionally evaluated with perplexity. However, this measure suffers from several shortcomings, in particular strong de- pendence on vocabulary size and lack of ability to directly evaluate scores assigned to malformed sentences. We address these deficiencies by pre- senting a new evaluation scheme for LM that sim- ulates an ASR setup. Rather than requiring the LM to produce a probability for a gold sentence, we instead present the LM with a set of alterna- tive sentences, including …

Content truncated to 500 words