Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4492 - 4507 July 10-15, 2022 ©2022 Association for Computational Linguistics Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer Yanpeng Zhao♣∗ Jack Hessel♥ Youngjae Yu♥ Ximing Lu♠♥ Rowan Zellers♠ Yejin Choi♠♥ ♣Institute for Language, Cognition and Computation, University of Edinburgh ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♥Allen Institute for Artificial Intelligence Abstract Machines that can represent and describe en- vironmental soundscapes have practical poten- tial, e.g., for audio tagging and captioning. Pre- vailing learning paradigms of audio-text con- nections have been relying on parallel audio- text data, which is, however, scarcely available on the web. We propose vip-AnT that induces Audio-Text alignment without using any par- allel audio-text data. Our key idea is to share the image modality between bi-modal image- text representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a tri- modal embedding space implicitly. In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2.2% R@1. We further investigate cases of minimal audio-text supervision, find- ing that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by 8% on US8K. How- ever, to match human parity on some zero-shot tasks, our empirical scaling experiments sug- gest that we would need about 221 ≈ 2M su- pervised audio-caption pairs. Our work opens up new avenues for learning audio-text connec- tions with little to no parallel audio-text data. 1 Introduction Environmental sound provides rich perspectives on the physical world. For example, if we hear: joyful laughing, a playful scream, and a splash; we not only can visualize literal objects / actions that might have given rise to the audio scene, but also, we can reason about plausible higher-level facets, e.g., a child speeding down a water slide at a water park, splashing through the water (see Figure 1). ∗Work was partially done during an internship at AI2. screaming and splashing people screaming on the water slide Figure 1: vip-AnT pivots audio and text via visual imagination. Machines capable of parsing, representing, and describing such environmental sound hold practical promise. For example, according to the National Association of the Deaf’s captioning guide, accessi- ble audio caption generation systems should go be- yond speech recognition (i.e., identifying speakers and transcribing the literal content of their speech) and provide the textual description of all the sound effects, e.g., “a large group of people talking excit- edly at a party”, in order to provide the full infor- mation contained in that audio.1 The dominant paradigm for studying machine hearing (Lyon, 2010) has been through human- annotated audio-text data, where text is either free-form audio descriptions (e.g., …

Content truncated to 500 words