Semi-supervised Development of ASR Systems for Multilingual Code-switched Speech in Under-resourced Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3468–3474 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3468 Semi-supervised Development of ASR Systems for Multilingual Code-switched Speech in Under-resourced Languages Astik Biswas1, Emre Yılmaz2, Febe de Wet1, Ewald van der Westhuizen1, Thomas Niesler1 1 Department of Electrical and Electronic Engineering, Stellenbosch University, South Africa 2 Department of Electrical and Computer Engineering, National University of Singapore, Singapore {abiswas, fdw, ewaldvdw, trn}@sun.ac.za, emre@nus.edu.sg Abstract This paper reports on the semi-supervised development of acoustic and language models for under-resourced, code-switched speech in five South African languages. Two approaches are considered. The first constructs four separate bilingual automatic speech recognisers (ASRs) corresponding to four different language pairs between which speakers switch frequently. The second uses a single, unified, five-lingual ASR system that represents all the languages (English, isiZulu, isiXhosa, Setswana and Sesotho). We evaluate the effectiveness of these two approaches when used to add additional data to our extremely sparse training sets. Results indicate that batch-wise semi-supervised training yields better results than a non-batch-wise approach. Furthermore, while the separate bilingual systems achieved better recognition performance than the unified system, they benefited more from pseudo-labels generated by the five-lingual system than from those generated by the bilingual systems. Keywords: Code-switched speech, under-resourced languages, multilingual speech, semi-supervised training, ASR 1. Introduction Much research has already been dedicated to the devel- opment of automatic speech recognition (ASR) for code- switching between various languages. English-Mandarin has probably been studied most extensively (Li and Fung, 2013a; Li and Fung, 2013b; Zeng et al., 2018; Vu et al., 2012; Taneja et al., 2019), but other language pairs such as Frisian-Dutch (Yılmaz et al., 2018c), Hindi-English (Pandey et al., 2018; Emond et al., 2018), English-Malay (Ahmed and Tan, 2012) and French-Arabic (Amazouz et al., 2017) have also received some attention. Developing models that are robust to the additional complexity asso- ciated with code-switching is challenging. The task be- comes even more difficult when the languages in question are under-resourced since small text and acoustic data sets limit modelling capacity. South Africa has a multilingual population of 57 million people and 11 official languages, including English. Due to the variety of geographically co-located languages, code- switching - the alternation between languages during com- munication - is a common phenomenon. Code-switching is most prevalent between English, a highly-resourced lan- guage, and the South African Bantu languages, which are all under-resourced. A corpus of code-switched speech originating from South African soap operas has recently been compiled to enable the development of ASR for this type of speech (van der Westhuizen and Niesler, 2018). Previous work demonstrated that multilingual training us- ing in-domain soap opera code-switched speech (Biswas et al., 2018a; Yılmaz et al., 2018a) and poorly matched mono- lingual South African speech (Biswas et al., 2018b) im- proves the performance of both bilingual and five-lingual ASR systems when the additional training data is from a closely-related language. Specifically, isiZulu, isiX- hosa, Sesotho and Setswana …

Content truncated to 500 words