A Unified Speaker Adaptation Approach for ASR

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9339–9349 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9339 A Unified Speaker Adaptation Approach for ASR Yingzhu Zhao1,2∗, Chongjia Ni2 , Cheung-Chi Leung2 Shafiq Joty1, Eng Siong Chng1 , Bin Ma2 1Nanyang Technological University, Singapore 2Machine Intelligence Technology, Alibaba Group {srjoty, aseschng}@ntu.edu.sg {yingzhu.zhao, ni.chongjia, cc.leung, b.ma}@alibaba-inc.com Abstract Transformer models have been used in auto- matic speech recognition (ASR) successfully and yields state-of-the-art results. However, its performance is still affected by speaker mis- match between training and test data. Fur- ther finetuning a trained model with target speaker data is the most natural approach for adaptation, but it takes a lot of compute and may cause catastrophic forgetting to the ex- isting speakers. In this work, we propose a unified speaker adaptation approach consist- ing of feature adaptation and model adaptation. For feature adaptation, we employ a speaker- aware persistent memory model which gener- alizes better to unseen test speakers by mak- ing use of speaker i-vectors to form a persis- tent memory. For model adaptation, we use a novel gradual pruning method to adapt to tar- get speakers without changing the model archi- tecture, which to the best of our knowledge, has never been explored in ASR. Specifically, we gradually prune less contributing param- eters on model encoder to a certain sparsity level, and use the pruned parameters for adap- tation, while freezing the unpruned parame- ters to keep the original model performance. We conduct experiments on the Librispeech dataset. Our proposed approach brings rel- ative 2.74-6.52% word error rate (WER) re- duction on general speaker adaptation. On target speaker adaptation, our method outper- forms the baseline with up to 20.58% relative WER reduction, and surpasses the finetuning method by up to relative 2.54%. Besides, with extremely low-resource adaptation data (e.g., 1 utterance), our method could improve the WER by relative 6.53% with only a few epochs of training. 1 Introduction End-to-end models yield state-of-the-art perfor- mance on automatic speech recognition (ASR) ∗Yingzhu Zhao is under the Joint PhD Program between Alibaba and Nanyang Technological University. in the past decade, such as connectionist tempo- ral classification (CTC) model (Miao et al., 2015; Graves, 2012), attention-based encoder-decoder model (Zhang et al., 2017), recurrent neural net- work transducer (RNN-T) (Graves, 2012), trans- former model (Dong et al., 2018) and conformer model (Gulati et al., 2020). However, model per- formance deteriorates due to speaker mismatch between training and test data. Given the target speaker, finetuning the trained model could alle- viate the speaker mismatch problem to some ex- tent, but finetuning the entire model requires large amounts of compute to be effective, and it could in turn bring catastrophic forgetting (McCloskey and Cohen, 1989) to the existing speakers. Currently, there are two lines of studies to ad- dress the speaker mismatch problem in neural net- work based models. One category is working on the acoustic features, i.e., either by normalizing acoustic features to be speaker-independent (Seide et al., 2011; …

Content truncated to 500 words