TAVT: Towards Transferable Audio-Visual Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14983–14999 July 9-14, 2023 ©2023 Association for Computational Linguistics TAVT:Towards Transferable Audio-Visual Text Generation Wang Lin∗, Tao Jin∗, Ye Wang∗, Wenwen Pan, Linjun Li, Xize Cheng, Zhou Zhao† Zhejiang University {linwanglw,jint_zju,yew}@zju.edu.cn {wenwenpan,lilinjun21,chengxize,zhaozhou}@zju.edu.cn Abstract Audio-visual text generation aims to under- stand multi-modality contents and translate them into texts. Although various transfer learn- ing techniques of text generation have been pro- posed, they focused on uni-modal analysis (e.g., text-to-text, visual-to-text) and lack consider- ation of multi-modal content and cross-modal relation. Motivated by the fact that humans can recognize the timbre of the same low-level concepts (e.g., footstep, rainfall, and laughing), even in different visual conditions, we aim to mitigate the domain discrepancies by audio- visual correlation. In this paper, we propose a novel Transferable Audio-Visual Text Genera- tion framework, named TAVT, which consists of two key components: Audio-Visual Meta- Mapper (AVMM) and Dual Counterfactual Contrastive Learning (DCCL). (1) AVMM first introduces a universal auditory semantic space and drifts the domain-invariant low-level con- cepts into visual prefixes. Then the reconstruct- based learning encourages the AVMM to learn “which pixels belong to the same sound” and achieve audio-enhanced visual prefix. The well- trained AVMM can be further applied to uni- modal setting. (2) Furthermore, DCCL lever- ages the destructive counterfactual transforma- tions to provide cross-modal constraints for AVMM from the perspective of feature distribu- tion and text generation. (3) The experimental results show that TAVT outperforms the state- of-the-art methods across multiple domains (cross-datasets, cross-categories) and various modal settings (uni-modal, multi-modal). 1 Introduction Audio-visual text generation bridges the gap be- tween perception (visual and auditory) and com- munication (via language), and is hence becoming an increasingly important goal for artificial agents. Uni-modal text generation tasks like machine trans- lation (Wang et al., 2022; Jin et al., 2022b; Yin ∗ Equal contribution. † Corresponding author Figure 1: Examples of transferable multi-modal text generation in the source domain (real-word) and the target domain (animation). For the same events (“speak- ing”, “cutting”), although the visual differences are sig- nificant, the sounds are similar. et al., 2022), and image caption (Chen et al., 2017; Tewel et al., 2021; Hu et al., 2022) have already flourished as a result of the large-scale pre-training and huge model capacity. However, for audio- visual text generation tasks, data annotation is more arduous (temporal structure) and expensive (re- quires monitors and speakers) than uni-modal text generation. Moreover, despite the effectiveness, existing works (Iashin and Rahtu, 2020a; Le et al., 2020; Hori et al., 2021) inevitably suffer severe degradation due to varying construction conditions in different domains. In this paper, to break through the constraint, we propose a novel task, named transferable audio- visual text generation. The main challenge of this task is the multi-modal domain shifts caused by varying conditions, like the visual style and audio energy. One common approach to handle the domain shift is domain-alignment-based trans- fer learning. However, existing works (Sun and Saenko, …

Content truncated to 500 words