Compact CNNs for End-to-End Keyword Spotting on Resource-Constrained Edge AI Devices

Compact CNNs for End-to-End Keyword Spotting on Resource-Constrained Edge AI Devices Joseph Lin Hsinchu County American School Hsinchu, Taiwan lintinghaojoseph@gmail.com Ren-yuan Lyu Department of Computer Science and Information Engineering Chang Gung University Taoyuan, Taiwan renyuan.lyu@gmail.com Abstract In this paper, we explore compact convolutional neural networks (CNNs) for end-to-end key- word spotting from raw audio to final recogni- tion results, without using traditional feature extraction based on spectrogram. Such fully CNN models reach 90.5% accuracy, an im- provement of 12.15% over traditional meth- ods with similar structures, which only achieve 78.35% accuracy, on the Speech Commands dataset. This shows that learned CNN fea- tures outperform predefined FFT-based trans- forms. The results show that compact end-to- end CNNs enable efficient, accurate small vo- cabulary keyword spotting that is well-suited for resource-constrained edge devices. All code will be released on the GitHub of the authors [Lin and Lyu, 2023]. Keywords: End-to-end models, raw audio pro- cessing, keyword spotting 1 Introduction Keyword spotting is a critical technology for edge applications such as voice-activated devices, smart speakers, industrial automation, security surveil- lance, and virtual assistants [Hoy, 2018]. It allows users to interact with devices hands-free, control devices without remote controls, operate machinery efficiently and safely, detect and respond to emer- gencies quickly, and get help and information from virtual assistants without opening dedicated apps. This technology has seen significant advances with the rise of deep learning. Recently, an open-sourcing neural net called Whisper from OpenAI, which approaches human- level robustness and accuracy in English and mul- tilingual speech recognition, has been released. [Radford, 2021] The Whisper model has proven to be very successful in large vocabulary speech recognition tasks. However, when it comes to keyword spotting on resource-constrained devices, the Whisper model has some disadvantages com- pared to a CNN-based model [Dai, 2016] due to its model complexity, inference speed/latency, and large memory footprint. This can be problematic on devices with limited computational and storage capacity. Recent research [Petrov, 2023] further shows that language model tokenizers favor En- glish over other languages, exacerbating the re- source constraints of non-English edge applica- tions. Convolutional neural networks (CNNs), on the other hand, have been proven particularly effective for speech processing tasks compared to older sta- tistical models [LeCun, 2015]. However, CNNs may quickly lose their advantages if the network is too deep [He, 2016]. In this paper, we explore various CNN archi- tectures for an end-to-end keyword-spotting appli- cation. Specifically, we investigate replacing the commonly used Fourier transform preprocessing with learned convolution layers for directly process- ing raw audio input. We also examine the tradeoffs between smaller and larger-scale CNN models in terms of accuracy and overfitting. Our contributions are three-fold. First, we demonstrate competitive accuracy with an end-to- end CNN model operating directly on raw audio data, removing the need for engineered feature ex- traction. Second, we show that smaller CNN mod- els can approach the accuracy of larger counter- parts, reducing overfitting concerns given the lim- ited training data size. Finally, we identify critical design …

Content truncated to 500 words