Cross-Modal Discrete Representation Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3013 - 3035 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Cross-Modal Discrete Representation Learning Alexander H. Liu SouYoung Jin Cheng-I Jeff Lai Andrew Rouditchenko Aude Oliva James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139, USA {alexhliu, souyoung, clai24, roudi, oliva, glass}@mit.edu Abstract In contrast to recent advances focusing on high- level representation learning across modalities, in this work we present a self-supervised learn- ing framework that is able to learn a represen- tation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantiza- tion that is shared across different modalities. Beyond the shared embedding space, we pro- pose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. We show that the proposed discretized multi-modal fine- grained representation (e.g., pixel/word/frame) can complement high-level summary represen- tations (e.g., video/sentence/waveform) for im- proved performance on cross-modal retrieval tasks. We also observe that the discretized rep- resentation uses individual clusters to represent the same semantic concept across modalities. 1 Introduction Toddlers acquire much of their knowledge through grounded learning – visual concepts can be ac- quired through language, and language acquisition emerges through visual interaction. Inspired by this type of grounded learning, a rich body of rep- resentation learning research (Harwath et al., 2018; Miech et al., 2020; Alayrac et al., 2020; Monfort et al., 2021; Luo et al., 2021) has been exploring the potential to learn from multi-modal data such as video-text, video-audio, and image-audio pairs. These works typically focus on learning a joint embedding space between different modalities, in which high-level summary representations are ex- tracted as embedding vectors. These embedding vectors often represent entire video clips, spoken utterances, or sentences as single vectors, and can be useful on tasks such as cross-modal data re- trieval, e.g., finding the most similar visual scene according to a spoken language description. The predominant approach to learning these embedding vectors is to use modality-independent encoders, and while this has been successful for downstream retrieval tasks, it makes it difficult to compare the activations of the encoders from different modali- ties. Further, the space of continuous embedding vectors is unbounded, which makes interpreting the learned representations challenging. To this end, we propose to jointly learn high- level embedding vector representations with a fine- grained discrete embedding space that is shared across different modalities. The discrete embed- ding space enables model interpretability since there are a finite number of embedding vectors which are shared across modalities. Besides the shared embedding space, we propose a Cross- Modal Code Matching (CMCM) objective that guides the embedding space to capture cross-modal correspondences of concepts, actions, and words. …

Content truncated to 500 words