Automatic Video Dubbing at AppTek

Automatic Video Dubbing at AppTek Mattia Di Gangi, Nick Rossenbach, Alejandro P´erez, Parnia Bahar Eugen Beck, Patrick Wilken, Evgeny Matusov AppTek GmbH Aachen, Germany mdigangi@apptek.com Abstract Automatic Video Dubbing is the process of automatically revoicing a video with a new script to make it accessible to a new audi- ence. In this paper, we describe AppTek Dubbing, a product that will be available in Q3 2022 to automatically dub a video into a target language. We plan multiple releases of the product with incremental features, as well as the possibility to allow human intervention for increased quality. 1 Introduction Video dubbing is the activity of revoicing a video while offering a viewing experience equivalent to the original video. The revoicing usually comes with a new script, and it should reproduce the orig- inal emotions, coherent with the body language, and be lip synchronized. ¨Oktem et al. (2019) and Federico et al. (2020) introduced two automatic dubbing systems as a cascade of automatic speech recognition (ASR), machine translation (MT) and Text-to-Speech (TTS), enhanced with a prosodic alignment (PA) component to transfer prosody through the pipeline. In this project, we aim to build an AD system in two phases: (1) voice- over; (2) full dubbing, and enhance it with human- in-the-loop capabilities for a higher quality. The product will be released in the form of REST APIs and a web interface in Q3 of the current year. The pricing will follow a pay-per-use scheme, with possible variations according to requested quality control or if the script to dub is provided by the user for higher quality dubbing. © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 2 Current Features Our current system is designed as an enhanced pipeline of ASR, MT and TTS. Our ASR sys- tem includes speaker diarization (the task of de- tecting “who speaks when”) so that consecutive segments from the same speaker can be translated as coherent units, and each speaker is assigned a unique voice. Our MT system is a Transformer- based encoder-decoder, augmented with metadata features for style adaptation (Matusov et al., 2020) and output length control (Lakew et al., 2019). The translations are performed from and to subtitle files to preserve the timestamps and use them as bound- aries for the synthesized voices. Additionally, we use speaker-adaptive TTS to reproduce the voice features of the original actor for the given seg- ment in the new language. Finally, the background sound, obtained via source separation, is merged with the synthesized voices for the final audio and video rendering. This system can already trans- late video contents and dub the output videos in a voice-over style. 3 Voice-over Voice-over is a simpler solution than dubbing, where the original voice’s volume is lowered down, and the new voice is rendered with a nat- ural volume over it, usually with a delay of some frames. Our system is already capable of perform- ing voice-over for some language pairs1 …

Content truncated to 500 words