From Speech-to-Speech Translation to Automatic Dubbing

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 257–264 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 257 From Speech-to-Speech Translation to Automatic Dubbing Marcello Federico Robert Enyedi Roberto Barra-Chicote Ritwik Giri Umut Isik Arvindh Krishnaswamy Hassan Sawaf∗ Amazon Abstract We present enhancements to a speech-to- speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, au- dio rendering to enriches text-to-speech out- put with background noise and reverberation extracted from the original audio. We report and discuss results of a first subjective evalua- tion of automatic dubbing of excerpts of TED Talks from English into Italian, which mea- sures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement. 1 Introduction Automatic dubbing can be regarded as an exten- sion of the speech-to-speech translation (STST) task (Wahlster, 2013), which is generally seen as the combination of three sub-tasks: (i) transcrib- ing speech to text in a source language (ASR), (ii) translating text from a source to a target language (MT) and (iii) generating speech from text in a tar- get language (TTS). Independently from the im- plementation approach (Weiss et al., 2017; Waibel, 1996; Vidal, 1997; Metze et al., 2002; Nakamura et al., 2006; Casacuberta et al., 2008), the main goal of STST is producing an output that reflects the linguistic content of the original sentence. On the other hand, automatic dubbing aims to replace all speech contained in a video document with speech in a different language, so that the result sounds and looks as natural as the original. Hence, in addition to conveying the same content of the original utterance, dubbing should also match the ∗ Contribution while the author was with Amazon. original timbre, emotion, duration, prosody, back- ground noise, and reverberation. While STST has been addressed for long time and by several research labs (Waibel, 1996; Vidal, 1997; Metze et al., 2002; Nakamura et al., 2006; Wahlster, 2013), relatively less and more sparse efforts have been devoted to automatic dubbing (Matou˘sek et al., 2010; Matou˘sek and V´ıt, 2012; Furukawa et al., 2016; ¨Oktem et al., 2019), al- though the potential demand of such technology could be huge. In fact, multimedia content cre- ated and put online has been growing at expo- nential rate, in the last decade, while availability and cost of human skills for subtitling and dub- bing still remains a barrier for its diffusion world- wide.1 Professional dubbing (Mart´ınez, 2004) of a video file is a very labor intensive process that in- volves many steps: (i) extracting speech segments from the audio track and annotating these with speaker information; (ii) transcribing the speech segments, (iii) translating the transcript in the tar- get language, (iv) adapting the translation for tim- ing, (v) casting the voice talents, (vi) perform- ing the dubbing sessions, …

Content truncated to 500 words