Length-Aware NMT and Adaptive Duration for Automatic Dubbing

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 138–143 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Length-Aware NMT and Adaptive Duration for Automatic Dubbing Zhiqiang Rao, Hengchao Shang, Jinlong Yang, Daimeng Wei, Zongyao Li, Jiaxin Guo, Shaojun Li, Zhengzhe Yu, Zhanglin Wu, Yuhao Xie, Bin Wei, Jiawei Zheng, Lizhi Lei and Hao Yang Huawei Translation Service Center, Beijing, China {raozhiqiang,shanghengchao,yangjinlong7,weidaimeng,lizongyao, guojiaxin1,lishaojun18,yuzhengzhe,wuzhanglin2,xieyuhao2,weibin29, zhengjiawei15,leilizhi,yanghao30}@huawei.com Abstract This paper presents the submission of Huawei Translation Services Center for the IWSLT 2023 dubbing task in the unconstrained set- ting. The proposed solution consists of a Transformer-based machine translation model and a phoneme duration predictor. The Trans- former is deep and multiple target-to-source length-ratio class labels are used to control tar- get lengths. The variation predictor in Fast- Speech2 is utilized to predict phoneme dura- tions. To optimize the isochrony in dubbing, re- ranking and scaling are performed. The source audio duration is used as a reference to re-rank the translations of different length-ratio labels, and the one with minimum time deviation is preferred. Additionally, the phoneme duration outputs are scaled within a defined threshold to narrow the duration gap with the source audio. 1 Introduction Automatic dubbing (AD) (Federico et al., 2020; Brannon et al., 2022; Chronopoulou et al., 2023) technology uses artificial intelligence (AI) to auto- matically generate dubbed audio for video content. Dubbing is the process of replacing the audio with a translation of the original audio in a different language. AI dubbing technology automates this process by using machine learning algorithms to translate the original audio and synthesize a new voice that sounds natural and resembles a human voice. The synthesized voice is then synchronized with the lip movements of the characters in the video to produce dubbed audio. This technology has the potential to significantly reduce the time and cost of creating dubbed audio and make it eas- ier to reach a global audience by translating video content into multiple languages. Recent advances in the field of automatic dub- bing have contributed to the development of more efficient and cost-effective methods for producing localized content. Researchers have utilized var- ious techniques and technologies, including ma- chine translation (MT) (Lopez, 2008; Vaswani et al., 2017), speech synthesis (Wang et al., 2017b; Ren et al., 2022), and speech recognition (Gulati et al., 2020; Schneider et al., 2019), to improve the accuracy and quality of automatic dubbing systems. NMT Variation Predictor Re-ranking & Scaling Source Audio Source Text Pause Alignment Target 1 Target 2 Target 3 Target 4 Target 5 <Equal> <Xlonger> <Longer> <Shorter> <Xshorter> Segmented Source Figure 1: System pipeline. Isometric machine translation (Lakew et al., 2022; Li et al., 2022) is a technique used in au- tomatic dubbing where translations should match a given length to allow for synchronicity between source and target speech. For neural MT, generat- ing translations of length close to the source length, while preserving quality is a challenging task. Con- trolling MT output length comes at a cost to trans- lation …

Content truncated to 500 words