Recycle Your Wav2Vec2 Codebook: A Speech Perceiver for Keyword Spotting

Proceedings of the 29th International Conference on Computational Linguistics, pages 7166–7170 October 12–17, 2022. 7166 Recycle Your Wav2Vec2 Codebook: a Speech Perceiver for Keyword Spotting Guillermo Cámbara TALN Research Group Universitat Pompeu Fabra Barcelona, Spain guillermo.cambara@upf.edu Jordi Luque Telefónica Research Telefónica I+D Barcelona, Spain jordi.luque@telefonica.com Mireia Farrús Centre de Llenguatge i Computació UBICS, Universitat de Barcelona Barcelona, Spain mfarrus@ub.edu Abstract Speech information in a pretrained wav2vec2.0 model is usually leveraged through its encoder, which has at least 95M parameters, being not so suitable for small footprint Keyword Spot- ting. In this work, we show an efficient way of profiting from wav2vec2.0’s linguistic knowl- edge, by recycling the phonetic information encoded in its latent codebook, which has been typically thrown away after pretraining. We do so by transferring the codebook as weights for the latent bottleneck of a Keyword Spot- ting Perceiver, thus initializing such model with phonetic embeddings already. The Perceiver design relies on cross-attention between these embeddings and input data to generate better representations. Our method delivers accuracy gains compared to random initialization, at no latency costs. Plus, we show that the phonetic embeddings can easily be downsampled with k-means clustering, speeding up inference in 3.5 times at only slight accuracy penalties. 1 Introduction Keyword Spotting (KWS) has benefited recently from the adoption of the Transformer architecture (Vaswani et al., 2017), as well as from recent ad- vances in self-supervised learning proposals like wav2vec2.0 (Baevski et al., 2020). Transformers are capable of capturing informa- tion from broader contexts, going beyond the lo- cality of convolutional neural networks (CNN) (LeCun et al., 1989) and avoiding the vanish- ing/exploding gradients from recurrent neural net- works (RNN) (Rumelhart et al., 1985). However, this comes at the quadratic cost of the self-attention mechanism (Bahdanau et al., 2014), which is even more pronounced in high-dimensional modalities like speech or vision. KWS models like the Key- word Spotting Transformer (KWT) (Berg et al., 2021) and the Audio Spectrogram Transformer (AST) (Gong et al., 2021), minimize such cost by downsampling the spectrogram space into patches, inspired by the Vision Transformer (ViT) (Dosovit- skiy et al., 2020) proposal from computer vision. In parallel, approaches like Wav2KWS (Seo et al., 2021) or the model from SUPERB (Yang et al., 2021) have successfully applied wav2vec2.0 to KWS. During training, wav2vec2.0 learns a la- tent codebook that codifies phonetic information, using each code as a target for training a feature en- coder. This codebook is typically thrown away after training, only keeping the encoder for downstream tasks like automatic speech recognition (ASR) or KWS. Even though the encoder is capable of ex- tracting rich features from raw waveforms, the size of it (at least 95M parameters for the BASE model) and its added latency time might discour- age straightforward use for small footprint KWS classifiers. In this short paper, we focus on exploring meth- ods for recycling the phonetic information from the wav2vec2.0 latent codebook, showing that such in- formation kickstarts the accuracy of a KWS model at initialization and …

Content truncated to 500 words