Improving Low-Resource Speech Recognition through Multilingual Fine-Tuning with Language Identifiers and Self-Training

Improving Low-Resource Speech Recognition through Multilingual Fine-Tuning with Language Identifiers and Self-Training Karol Nowakowski Tohoku University of Community Service and Science karol@koeki-u.ac.jp Michal Ptaszynski Kitami Institute of Technology michal@mail.kitami-it.ac.jp Abstract Previous work has demonstrated that multilin- gual fine-tuning of a pretrained multilingual speech representation model can lead to im- proved speech recognition accuracy when there is extremely little target language data avail- able. In this paper we show that fine-tuning on labeled speech data from multiple languages sharing common phonological traits, prepro- cessed by attaching a language identifier to each speech sample, yields competitive results compared to monolingual fine-tuning, even if a moderate amount of target language data is available. In order to further improve the per- formance of our system, we apply self-training using unlabeled speech data. Our results in- dicate that fine-tuning a speech recognition model jointly on a combination of multilin- gual data and pseudo-labeled data yields su- perior performance compared to using any of the two augmentation techniques individually. We also find that models fine-tuned on multi- lingual data with language identifiers produce better results even if explicit information about language identity is not provided at inference time. Keywords: Speech recognition, Under- resourced language, Ainu, Multilingual learning, Transfer learning, Cross-lingual transfer, Language identifiers, Self-training 1 Introduction It is believed that speech processing technolo- gies can be leveraged in language documentation projects to speed up labor-intensive tasks such as speech transcription. However, for many languages it is difficult to develop a speech recognition system useful in real-world applications, as the accuracy of current machine learning-based methods in a low- data scenario still lags behind, compared to lan- guages with ample training data available. In order to push forward the development of low-resource speech recognition, previous studies have proposed various data augmentation techniques – such as self- training (Synnaeve et al., 2020; Xu et al., 2020) – transfer learning utilizing speech representations learned in unsupervised manner from raw speech data (Schneider et al., 2019; Baevski et al., 2020; Hsu et al., 2021), and cross-lingual transfer meth- ods (Toshniwal et al., 2018; Conneau et al., 2021). It has been shown that pretraining speech represen- tations jointly on unlabeled speech data in multiple languages results in models with better downstream performance for low-resource languages than train- ing on each language individually (Conneau et al., 2021), especially if data from related languages is available in relatively large amounts. Recently, Nowakowski et al. (2023) found that the benefits of cross-lingual transfer to an under-resourced lan- guage from similar speech varieties also extend to supervised fine-tuning, if there is very little (less than 1 hour) labeled data in the target language available. If a speech recognition model is trained on data in multiple languages simultaneously and only pro- vided with the acoustic features of speech samples as input, it must implicitly learn to distinguish be- tween different languages appearing in the training data in order to be able to produce a correct output, which can be particularly challenging in low-data scenarios. This requirement …

Content truncated to 500 words