Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing William Brannon∗ MIT Media Lab, USA wbrannon@mit.edu Yogesh Virkar AWS AI Labs, USA yvvirkar@amazon.com Brian Thompson† AWS AI Labs, USA brianjt@amazon.com Abstract We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 profession- ally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (char- acter length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as transfer of semantic properties such as emphasis and emotion, in automatic dubbing systems. 1 Introduction Considerable attention has been paid to the dub- bing of video content from one language into another, both in the literature of several disciplines and in the daily practice of the entertainment in- dustry. One influential line of work, in the fields of film studies and audiovisual translation, stud- ies human dubbing from a qualitative perspective (Chaume, 2012; Zabalbeascoa, 1997, 2008; Freddi and Pavesi 2009), as a profession and semiotic ac- tivity. This literature has developed a rich body of theory on the nature of the human dubbing task, and the ways humans approach it, but has little engagement with large-scale data. More re- cently, machine-learning practitioners have taken up the task of building multimodal systems for au- tomatic dubbing (e.g., Saboo and Baumann, 2019; Federico et al., 2020a; Tam et al., 2022), but lack ∗Work conducted during an internship at Amazon. †Corresponding author. deep empirical or theoretical bases for how to organize their work. What is missing from both literatures, and can help bridge the gap between them, is a large-scale study of human dubbing in practice: a data-driven examination of the way humans actually perform this task. Such an analysis can have benefits for both the qualitative study of human dubbing, by providing empirical evidence of how dubbing teams approach their work, and informing fu- ture machine-learning work on automatic dubbing systems. It is exactly this analysis we undertake in this work. Human dubbing involves a sequence of human contributors each with control over a different as- pect of the process (Chiaro, 2008a,b; Matamala, 2010; Chaume, 2012). The first step is an approx- imately literal translation of the original script, done by a dialogue translator. Next, a dialogue adaptor will modify this translation into a plausi- ble script meeting the requirements for dubbing such as isochrony, lip sync, kinesic synchrony, and so on. Finally, the translated and modified script will go to a …

Content truncated to 500 words