WhisperHakka: A Hybrid Architecture Speech Recognition System for Low-Resource Taiwanese Hakka

WhisperHakka: A Hybrid Architecture Speech Recognition System for Low-Resource Taiwanese Hakka Ming-Hsiu Chiang Chien-Hung Lai Hsuan-Sheng Chiu Advanced Technology Laboratory, Telecommunication Laboratories, Chunghwa Telecom Co., Ltd., Taiwan {kurt, jhlai, samhschiu}@cht.com.tw Abstract Deep learning-based automatic speech recognition (ASR) design has been grow- ing in popularity. Besides the ASR model depends on the reliable speech representa- tion offered by the self-supervised learn- ing (SSL) model, Whisper is also a pow- erful model that makes use of the knowl- edge from large-scale labeled datasets and the self-attention mechanism. However, its inherent training method decreases the potential to expand on low-resource lan- guage because of the difficulty of acquir- ing labeled data. As a result, we inte- grated both the Wav2vec2 and Whisper into an ASR system not only to provide ex- tra abundant information on features but also to have the ability to train on unla- beled data through the SSL model while retaining the capacity of Whisper. Experi- mental Results show that the proposed hy- brid architecture system outperforms the vanilla Whisper in the reading speech sce- nario, achieving a roughly 21% improve- ment in recognition rate. Keywords: ASR, Whisper, Wav2vec2, Taiwanese Hakka, Self-Supervised Learning 1 Introduction With the development of technology in full swing, traditional interaction interfaces have gradually progressed from keyboards, mice, and touchscreens to a new generation of in- teraction interfaces based on natural conver- sation. Both businesses and individuals are attempting to use such technology to enhance the convenience and comfort of people’s lives. Speech recognition is unquestionably one of the most important technologies used in the current process of natural conversation interac- tion. Typically, the system needs to transcribe the user’s speech into text before applying nat- ural language processing to the text. Due to the significance of ASR, the technology’s ad- vancement over the past ten years has been extremely rapid. In particular, the introduc- tion of deep learning (DL) has resulted in a significant decrease of more than 50% in the word error rate in common languages (Prab- havalkar et al., 2023). Sadly, despite the advances in technology, the effectiveness of speech recognition systems is still constrained by the sheer amount and diversity of training data. As a result, only roughly 100 languages are currently covered by ASR models, compared to the more than 7,000 known languages in the world. Specifically, even Hakka, which is spoken by the third- largest population in Taiwan after Chinese and Hokkien, is nevertheless listed by the United Nations Educational, Scientific and Cultural Organization (UNESCO) as a severely endan- gered level. Language is not just a means of communication, it also plays a significant role in preserving cultural legacy. Therefore, it can be claimed that language is a valuable resource for people, which is why it is cru- cial to protect endangered languages. From a digital standpoint, it is believed that AI tech- nology can benefit these endangered languages through speech synthesis, which can turn text into conversational speech, and speech recogni- tion, which can extract text from speech to …

Content truncated to 500 words