Code-Switch Language Model with Inversion Constraints for Mixed Language Speech Recognition

Proceedings of COLING 2012: Technical Papers, pages 1671–1680, COLING 2012, Mumbai, December 2012. Code-switch Language Model with Inversion Constraints for Mixed Language Speech Recognition Ying Li, Pascale Fung Human Language Technology Center Department of Electronic & Computer Engineering, HKUST eewing@ust.hk, pascale@ee.ust.hk ABSTRACT We propose a first ever code-switch language model for mixed language speech recognition that incorporates syntactic constraints by a code-switch boundary prediction model, a code-switch translation model, and a reconstruction model. A WFST-based decoder then recognizes speech by combining an acoustic model, a pronunciation model and the code-switch language model in an integrated approach. Our proposed approach avoids making early decisions on code-switch boundaries and is therefore more robust than previous approaches. Our proposed system using the code-switch language model outperforms a baseline of interpolated language models by a statistically significant 0.91% on a mixed language lecture speech corpus, and 1.25% on a mixed language lunch conversation corpus. Our method also outperforms a language model that permits code-switch at all word boundaries by a statistically significant 1.35% on the lecture speech corpus and 1.69% on the lunch conversation corpus. KEYWORDS: Code-switch, mixed language, language modeling. 1671 1 Introduction Multilingual people often code-switch (CS) — mixing two languages in the same sentence (intra- sentential code-switch) or between sentences (inter-sentential code-switch). For inter-sentential switches, many researchers use two language models to decode separate sentences (Fugen et al., 2003). However, a sentence that contains two languages poses a more formidable challenge to speech recognition systems, as the same sentence would contain words or phrases belonging to two or more grammatical systems or subsystems (Gumperz, 1982). It is challenging to predict where in the sentence the speaker switches to another language and back, if at all. Some researchers (Lee et al., 2009) use the transcription of the code-switch speech to train a domain- dependent language model. This approach is limited by the small amount of code-switch data available for training. Code-switch should be distinguished from loanword, which is a word borrowed from one language and incorporated into another language to become part of the lexicon. Code-switch speech is where the speaker actually tries to speak another language in its own grammar. In code-switch, the matrix language is the ‘principal’ language, where the ‘embedded’ language is the second language(C. and C., 1993; Coulmas, 1998). There are two main approaches to recognizing code-switch mixed language speech. One is to detect the boundaries at which the speaker code-switches, then identify the language in the speech segments between the boundaries, and decode the speech segments using the acoustic and language models in the corresponding language(Chan et al., 2005; Shia et al., 2004; Lyu and Lyu, 2008; Vu et al., 2012). For text only code-switch language, Solorio and Liu (2008) used Naive Bayes and Value Feature Interval to classify the hypothesis code-switch points by F-measure and the naturalness rating. However, this approach requires multiple passes of boundary detection, language identification and speech recognition. The boundaries and language identity of each speech segment are irreversibly determined by the …

Content truncated to 500 words