Integration of Dubbing Constraints into Machine Translation

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 94–101 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 94 Integration of Dubbing Constraints into Machine Translation Ashutosh Saboo BITS Pilani, K.K. Birla Goa Campus∗ Goa, India ashutosh.saboo96@gmail.com Timo Baumann Department of Informatics, Universit¨at Hamburg, Germany baumann@informatik.uni-hamburg.de Abstract Translation systems aim to perform a meaning- preserving conversion of linguistic material (typically text but also speech) from a source to a target language (and, to a lesser degree, the corresponding socio-cultural contexts). Dub- bing, i. e., the lip-synchronous translation and revoicing of speech adds to this constraints about the close matching of phonetic and re- sulting visemic synchrony characteristics of source and target material. There is an inher- ent conflict between a translation’s meaning preservation and its ‘dubbability’ and the re- sulting trade-off can be controlled by weigh- ing the synchrony constraints. We introduce our work, which to the best of our knowl- edge is the first of its kind, on integrating syn- chrony constraints into the machine translation paradigm. We present first results for the inte- gration of synchrony constraints into encoder decoder-based neural machine translation and show that considerably more ‘dubbable’ trans- lations can be achieved with only a small im- pact on BLEU score, and dubbability improves more steeply than BLEU degrades. 1 Introduction Dubbing, the lip-synchronous translation and re- voicing of audio-visual media, is essential for the full-fledged reception of foreign movies, TV shows, instructional videos, advertisements, or short so- cial media clips. Dubbing does not contend for the viewers’ visual attention like subtitles (D´ıaz-Cintas and Remael, 2014) do, and unlike voice-over or asynchronous speech there is no (or only little) mis- match of visual and auditory impression where the resulting cognitive dissonance would otherwise in- crease the viewers’ cognitive load, or even lead to understanding errors (McGurk and Macdonald, ∗*This work was performed during an internship at Uni- versit¨at Hamburg, Germany. 1976). Dubbing is still primarily studied in audio- visual translation (Orero, 2004; Chaume, 2012) and performed manually, unlike textual translation, which is largely being automated or supported by computer-aided translation (Koehn, 2009). Recent break-throughs in speech-to-speech trans- lation (Jia et al., 2019), do not yield translations that systematically observe dubbing constraints, i. e. do not match phonetically (or rather: visemically) the original source (we call this ‘dubbability’). It is our goal to create MT systems where the dub- bability of the translation can be controlled so as to optimize the trade-off between translation qual- ity and lip-synchrony of the dubbed speech. We hope that more widely available dubbing across languages will help to stimulate access to foreign media and foster inter-cultural exchange. We argue that dubbable MT will not simply emerge from training on dubbed audio-visual cor- pora, i. e. implicitly. By comparison, audio-visual corpora will always remain smaller than pure text- to-text translation corpora. As a result, merely rely- ing on training a conventional MT system on large amounts of dubbing texts is bound to severely limit performance. …

Content truncated to 500 words