Automatic Song Translation for Tonal Languages

Findings of the Association for Computational Linguistics: ACL 2022, pages 729 - 743 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Automatic Song Translation for Tonal Languages Fenfei Guo University of Maryland fguo1@umd.edu Chen Zhang Zhejiang University zc99@zju.edu.cn Zhirui Zhang Tencent AI Lab zrustc11@gmail.com Qixin He Purdue University heqixin@purdue.edu Kejun Zhang Zhejiang University zhangkejun@zju.edu.cn Jun Xie Alibaba DAMO Academy qingjing.xj@alibaba-inc.com Jordan Boyd-Graber CS, iSchool, UMIACS, LSC University of Maryland jbg@umiacs.umd.edu Abstract This paper develops automatic song transla- tion (AST) for tonal languages and addresses the unique challenge of aligning words’ tones with melody of a song in addition to conveying the original meaning. We propose three criteria for effective AST—preserving meaning, singa- bility and intelligibility—and design metrics for these criteria. We develop a new benchmark for English–Mandarin song translation and de- velop an unsupervised AST system, Guided AliGnment for Automatic Song Translation (GagaST), which combines pre-training with three decoding constraints. Both automatic and human evaluations show GagaST successfully balances semantics and singability. 1 Introduction Suppose you are asked to translate the lyrics “Let it go” from the Disney musical Frozen into Mandarin Chinese. Some good, literal translations of this would be A) “fàng shˇou”, B) “fàng shˇou ba” or C) “ràng t¯a qù ba” (Figure 1); these get the meaning across and are the domain of traditional machine translation. However, what if you needed to sing this song in Mandarin? These literal translations simply do not work: Translations A and C do not match the number of notes and break the original rhythm; while the tones of Translation B does not match the pitch flow of the original melody. Song translation, unlike translation lyrics for under- standing (subtitling), aims to translate the lyrics so that it can be sung with the original melody. There- fore, the translated lyrics must match the prosody of the pre-existing music in addition to retaining the original meaning. In Singable Translations of Songs, Low (2003) says, this is an uncommon Google Translate 放 手 吧 fàng shǒu ba 放 手 吧 fàng shǒu ba Human Lyrics translation Human Song translation 放 手 fàng shǒu 放 手 fàng shǒu 随 他 吧 suí tā ba 随 他 吧 suí tā ba Baidu Translate 让 它 去 吧 让 它 去 吧 ba ràng tā ba ràng tā qù qù Transition direction of successive notes/tones by pitch level: up , down Let it go_______________ Let it go_______________ Voice Figure 1: Example Mandarin translations for “Let it go” in Frozen. Of these, only the official human song translation is something a singer could actually sing: it fits the length of the notes and matches the tones with the pitch of notes. GagaST finds translations that satisfy these constraints. and an unusually complex task, a translator con- sider rhythm, notes’ pitches, phrasing, and stress. Nonetheless, there are cultural and commercial in- centives for more efficient song translation; Frozen alone made over a half a billion dollars in non- English box office receipts1 and the musical Les Misérables has been …

Content truncated to 500 words