Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4491 - 4503 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition Xichen Pan1, Peiyu Chen1, Yichen Gong2, Helong Zhou2, Xinbing Wang1, Zhouhan Lin1† 1Shanghai Jiao Tong University, 2Horizon Robotics {flash321, pietychen, xwang8}@sjtu.edu.cn, {yichen01.gong, helong.zhou}@horizon.ai lin.zhouhan@gmail.com Abstract Training Transformer-based models demands a large amount of data, while obtaining aligned and labelled data in multimodality is rather cost-demanding, especially for audio-visual speech recognition (AVSR). Thus it makes a lot of sense to make use of unlabelled uni- modal data. On the other side, although the effectiveness of large-scale self-supervised learning is well established in both audio and visual modalities, how to integrate those pre- trained models into a multimodal scenario re- mains underexplored. In this work, we suc- cessfully leverage unimodal self-supervised learning to promote the multimodal AVSR. In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual data into characters through a combination of CTC and seq2seq decoding. We show that both components inherited from unimodal self- supervised learning cooperate well, resulting in that the multimodal framework yields com- petitive results through fine-tuning. Our model is experimentally validated on both word-level and sentence-level tasks. Especially, even without an external language model, our pro- posed model raises the state-of-the-art perfor- mances on the widely accepted Lip Reading Sentences 2 (LRS2) dataset by a large margin, with a relative improvement of 30%. * 1 Introduction Audio-Visual Speech Recognition (AVSR) is a speech recognition task that leverages both an au- dio input of human voice and an aligned visual in- put of lip motions. It has been one of the successful application fields that involve multiple modalities †Corresponding author. *Our codes are available at https://github.com/L UMIA-Group/Leveraging-Self-Supervised-Le arning-for-AVSR. in recent years. Due to the limited amount of la- beled, multimodal aligned data and the difficulty of recognition from the visual inputs (i.e., lip reading), it is a challenging task to tackle. Existing AVSR models tend to use extra data to increase the performance of the system, in a form of inserting an extra supervised learning stage in the training process. For example, many existing meth- ods rely on an extra sequence level classification to bootstrap its learning on visual features. Petridis et al. (2018); Zhang et al. (2019) train their visual front-end on LRW (Chung and Zisserman, 2016) before learning on the AVSR task. Afouras et al. (2018a,b) chunks the MV-LRS data (Chung and Zisserman, 2017) into pieces of words and pre-train the model through classification. VoxCeleb (Chung et al., 2018) are also used in Afouras et al. (2020) for the same purpose. Learning an effective visual front-end could still be notoriously hard, even with these extra supervised learning tasks. Sometimes curriculum learning is required to adapt the learned visual front-end into AVSR task (Afouras et al., 2018a). End-to-end …

Content truncated to 500 words