Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages

Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages Onno Eberhard and Torsten Zesch Language Technology Lab University of Duisburg-Essen, Germany onno.eberhard@stud.uni-due.de torsten.zesch@uni-due.de Abstract In this paper, we investigate the effect of layer freezing on the effectiveness of model trans- fer in the area of automatic speech recogni- tion. We experiment with Mozilla’s Deep- Speech architecture on German and Swiss Ger- man speech datasets and compare the results of either training from scratch vs. transferring a pre-trained model. We compare different layer freezing schemes and find that even freezing only one layer already significantly improves results. 1 Introduction The field of automatic speech recognition (ASR) is dominated by research specific to the English lan- guage. There exist plenty available text-to-speech models pre-trained on (and optimized for) English data. When it comes to a low-resource language like Swiss German, or even standard German, only a very limited number of small-scale models is available. In this paper, we train Mozilla’s imple- mentation1 of Baidu’s DeepSpeech ASR architec- ture (Hannun et al., 2014) on these two languages. We use transfer learning to leverage the availability of a pre-trained English version of DeepSpeech and observe the difference made by freezing different numbers of layers during training. 2 Transfer Learning and Layer Freezing Deep neural networks can excel at many different tasks, but they often require very large amounts of training data and computational resources. To rem- edy this, it is often advantageous to employ transfer learning: Instead of initializing the parameters of the network randomly, the optimized parameters of a network trained on a similar task are reused. 1https://github.com/mozilla/DeepSpeech Those parameters can then be fine-tuned to the spe- cific task at hand, using less data and fewer compu- tational resources. In the fine-tuning process many parameters of the original model may be “frozen”, i.e. held constant during training. This can speed up training and improve results when less training data is available (Kunze et al., 2017). The idea of taking deep neural networks trained on large datasets and fine-tuning them on tasks with less available train- ing data has been popular in computer vision for years (Huh et al., 2016). More recently, with the emergence of end-to-end deep neural networks for automatic speech recognition (like DeepSpeech), it has also been used in this area (Kunze et al., 2017; Li et al., 2019). Deep neural networks learn representations of the input data in a hierarchical manner. The in- put is transformed into simplistic features in the first layers of a neural network and into more com- plex features in the layers closer to the output. If we assume the simplistic feature representations are applicable in similar, but different, contexts, layer-wise freezing of parameters seems like a good choice. This is further reinforced by findings from image classification (Yosinski et al., 2014), where the learned features can additionally be nicely vi- sualized (Zeiler and Fergus, 2014). As for automatic speech recognition, the repre- sentations learned by the layers is not as clear-cut …

Content truncated to 500 words