MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11610–11625 July 9-14, 2023 ©2023 Association for Computational Linguistics MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition Yuchen Hu1, Chen Chen1, Ruizhe Li2, Heqing Zou1, Eng Siong Chng1 1Nanyang Technological University, Singapore 2University of Aberdeen, UK {yuchen005@e., chen1436@e., heqing001@e., aseschng@}ntu.edu.sg, ruizhe.li@abdn.ac.uk Abstract Audio-visual speech recognition (AVSR) at- tracts a surge of research interest recently by leveraging multimodal signals to understand hu- man speech. Mainstream approaches address- ing this task have developed sophisticated ar- chitectures and techniques for multi-modality fusion and representation learning. However, the natural heterogeneity of different modali- ties causes distribution gap between their repre- sentations, making it challenging to fuse them. In this paper, we aim to learn the shared rep- resentations across modalities to bridge their gap. Different from existing similar methods on other multimodal tasks like sentiment analy- sis, we focus on the temporal contextual depen- dencies considering the sequence-to-sequence task setting of AVSR. In particular, we pro- pose an adversarial network to refine frame- level modality-invariant representations (MIR- GAN), which captures the commonality across modalities to ease the subsequent multimodal fusion process. Extensive experiments on pub- lic benchmarks LRS3 and LRS2 show that our approach outperforms the state-of-the-arts1. 1 Introduction Human perception of the world intrinsically com- prises multiple modalities, including vision, au- dio, text, etc. (McGurk and MacDonald, 1976; Bal- trušaitis et al., 2018). Audio-visual speech recog- nition (AVSR) leverages both audio and visual modalities to understand human speech, improving the noise-robustness of audio-only speech recog- nition with noise-invariant lip movement informa- tion (Sumby and Pollack, 1954). Thanks to recent advances of deep learning techniques, AVSR re- search has gained a remarkable progress (Afouras et al., 2018a; Ma et al., 2021; Shi et al., 2022b). Currently, the mainstream AVSR approaches are centered around developing sophisticated archi- tectures and techniques for multi-modality fusion, 1Code is available at https://github.com/YUCHE N005/MIR-GAN. Modality-invariant Modality-specific Fusion Frame-level Multimodal Representations Recognition frames Video stream Audio stream "I love life" Figure 1: Multimodal learning of frame-level modality- invariant and -specific representations. including simple feature concatenation (Makino et al., 2019; Ma et al., 2021; Pan et al., 2022), recur- rent neural network (Petridis et al., 2018; Xu et al., 2020) and cross-modal attention module (Afouras et al., 2018a; Lee et al., 2020). Despite the ad- vances, these approaches are often challenged by the representation gap persisting between naturally heterogeneous modalities (Hazarika et al., 2020). Recently in some other multimodal tasks like sentiment analysis (Hazarika et al., 2020; Yu et al., 2021; Yao and Mihalcea, 2022) and cross-modal retrieval (Xiong et al., 2020), there have been re- search works proposing to learn two distinct repre- sentations to benefit multimodal learning. The first representation is modality-invariant, where multi- ple modalities of a same utterance are mapped to a shared space, indicating the homogeneous seman- tic meaning from the speaker. In addition, they also learn modality-specific representations that are private to each modality. Given …

Content truncated to 500 words