OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6592–6607 July 9-14, 2023 ©2023 Association for Computational Linguistics OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment Xize Cheng1∗, Tao Jin1* , Linjun Li1* , Wang Lin1 , Xinyu Duan2 , Zhou Zhao1 † 1Zhejiang University ,2Huawei Cloud {chengxize,jint_zju,zhaozhou}@zju.edu.cn duanxinyu@huawei.com Abstract Speech Recognition builds a bridge between the multimedia streaming (audio-only, visual- only or audio-visual) and the corresponding text transcription. However, when training the specific model of new domain, it often gets stuck in the lack of new-domain utterances, es- pecially the labeled visual utterances. To break through this restriction, we attempt to achieve zero-shot modality transfer by maintaining the multi-modality alignment in phoneme space learned with unlabeled multimedia utterances in the high resource domain during the pre- training (Shi et al., 2022), and propose a train- ing system Open-modality Speech Recogni- tion (OpenSR) that enables the models trained on a single modality (e.g., audio-only) ap- plicable to more modalities (e.g., visual-only and audio-visual). Furthermore, we employ a cluster-based prompt tuning strategy to han- dle the domain shift for the scenarios with only common words in the new domain ut- terances. We demonstrate that OpenSR en- ables modality transfer from one to any in three different settings (zero-, few- and full- shot), and achieves highly competitive zero- shot performance compared to the existing few- shot and full-shot lip-reading methods. To the best of our knowledge, OpenSR achieves the state-of-the-art performance of word error rate in LRS2 on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respec- tively. The code and demo are available at https://github.com/Exgc/OpenSR. 1 Introduction Speech Recognition (Afouras et al., 2018a; Ren et al., 2021; Zhao et al., 2020) (e.g., Audio-Visual Speech Recognition known as AVSR) transcribs visual and audio data into text form, building a bridge between multi-media speech (Cheng et al., 2023; Huang et al., 2023a,b; Cui et al., 2022; Aox- iong et al., 2023) and natural language (Yin et al., ∗Equal contribution. †Corresponding author. Figure 1: Comparison between OpenSR and previous methods under different sizes of labeled visual utter- ance. As the results highlighted by the red circle show, OpenSR with zero-shot even outperforms most previous methods with full-shot. 2022, 2021; Jin et al., 2022a,b; Jin, 2021). Among them, ASR (Automatic Speech Recognition) and lip-reading (VSR, Visual Speech Recognition) are twin tasks transcribed using only audio and only vision, respectively. Audio utterances with clear pronunciation are sufficient for ASR training, most of which can be easily collected from recordings of telephone conversations and audiobooks (Korvas et al., 2014). While current lip-reading training systems require mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions around the lip area are clearly captured (Prajwal et al., 2022). The significant difficulty of collecting labeled visual utterances hinders the training of lip-reading models suitable for new do- mains or low-resource domains, resulting in the rel- atively low-speed development of domain-specific lip-reading models, compared with ASR models. Since audio …

Content truncated to 500 words