The Two Shades of Dubbing in Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4327–4333 Barcelona, Spain (Online), December 8-13, 2020 4327 The Two Shades of Dubbing in Neural Machine Translation Alina Karakanta1,2, Supratik Bhattacharya3, Shravan Nayak4, Timo Baumann5, Matteo Negri1, Marco Turchi1 1Fondazione Bruno Kessler, Trento, Italy 2University of Trento, Italy 3Birla Institute of Technology and Science, Pilani, India 4Indian Institute of Technology (BHU) Varanasi, India 5Universit¨at Hamburg, Germany {karakanta,negri,turchi}@fbk.eu, f20170745@pulani-bits-pilani.ac.in, pshravan.nayak.ece17@itbhu.ac.in, baumann@informatik.uni-hamburg.de Abstract Dubbing has two shades; synchronisation constraints are applied only when the actor’s mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depend- ing on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive fea- tures between on- and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease trans- lation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation. 1 Introduction Dubbing is a form of audiovisual translation (AVT) which consists in replacing the original audio track of a film dialogue with another track containing the dialogue translated in a different language. It is the preferred type of AVT in countries with large film and streaming markets, e.g. Germany, Italy, Spain and parts of Latin America (Bogucki and D´ıaz-Cintas, 2020). Dubbing, as a form of translation, is among the few where Machine Translation has found no steady ground yet. The first reason is the particularity of dubbing as a genre. Translation for dubbing is a text written to be spoken, therefore it should bear a close approximation to orality and reflect oral unlabored dialog (Chaume-Varela, 2006). However, dubbing is not spontaneous, impromptu speech but rather a carefully-prepared imitation of spoken language. This prefabricated orality (Ba˜nos-Pi˜nero and Chaume, 2009) is what confers dubbing particular linguistic characteristics. The second challenge lies in the constraints affecting the translation. The most distinctive character- istic of dubbing is the need for synchronisation when an actor’s mouth appears on screen. Isochrony requires that the duration of the source and target utterance is equal, in order for the translated dialogue to exactly fit the time during which the actor speaks. At a second level, lip-sync consists in adapting the translation to match the articulatory mouth movements of the actor, mainly matching open vowels and bilabials.1 For example, the sentence “I get strong off other people’s fear” is translated as “El miedo de los dem´as me da fuerzas”, instead of a more literal translation “Me fortalezco con el miedo de otras per- sonas”, in order to match the duration of the source utterance and the overlap of f between fear-fuerzas. From the above, it becomes evident that there is a clear dichotomy in dubbing strategies when an actor’s This work is licensed under a Creative Commons Attribution 4.0 International …

Content truncated to 500 words