AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8590–8604 July 9-14, 2023 ©2023 Association for Computational Linguistics AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation Rongjie Huang1∗, Huadai Liu1∗, Xize Cheng1∗, Yi Ren2, Linjun Li1, Zhenhui Ye1, Jinzheng He1, Lichao Zhang1, Jinglin Liu2, Xiang Yin2, Zhou Zhao1† Zhejiang University1, ByteDance2 {rongjiehuang,liuhuadai,zhaozhou}@zju.edu.cn {ren.yi,liu.jinglin,yinxiang.stephen}@bytedance.com Abstract Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AV- TranSpeech, the first audio-visual speech-to- speech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual in- formation to promote system robustness and opens up a host of practical applications: dic- tation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn con- textual representation, and 2) introduce cross- modal distillation with S2ST models trained on the audio-only corpus to further reduce the re- quirements of visual data. Experimental results on two language pairs demonstrate that AV- TranSpeech outperforms audio-only models un- der all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with base- lines.1 1 Introduction Speech-to-speech translation (S2ST) models (Tjan- dra et al., 2019; Zhang et al., 2020; Jia et al., 2021) relying on speech data have achieved high perfor- mance and significantly broken down communica- tion barriers between people not sharing a common language, which attracts broad interest in the ma- chine learning community (Huang et al., 2022c; Huang et al.). Among them, direct systems (Lee ∗Equal contributions †Corresponding author 1Audio samples are available at https:// AV-TranSpeech.github.io/. et al., 2021a,b; Huang et al., 2022d) leverage recent progress on self-supervised discrete units learned from unlabeled speech for building textless S2ST, further supporting translation between unwritten languages (Chen et al., 2022). As speech production (Huang et al., 2023; Lam et al., 2021; Huang et al., 2022b) is accompanied by the movement of lips and teeth, it can be visually interpreted to understand speech. In recent years, significant research (Shi et al., 2022a; Prajwal et al., 2022) has introduced joint modeling of spoken lan- guage and vision: Shi et al. (2022b) investigate to learn lip-based audio-visual speaker embeddings, where the speaker’s mouth area is used alongside speech as inputs. Chern et al. (2022) focus on audio-visual speech enhancement and separation which better integrates visual information. Despite their success, it is unclear how lip can contribute to audio-based S2ST, and how to incorporate vi- sual modality as auxiliary information in S2ST. A visual translator may open up a host of practi- cal applications: improving speech translation in noisy environments, enabling dictation, or dubbing archival films. Despite the benefits of audio-visual approaches, training …

Content truncated to 500 words