Unsupervised Melody-to-Lyrics Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9235–9254 July 9-14, 2023 ©2023 Association for Computational Linguistics Unsupervised Melody-to-Lyric Generation Yufei Tian1∗, Anjali Narayan-Chen2, Shereen Oraby2, Alessandra Cervone2, Gunnar Sigurdsson2, Chenyang Tao2, Wenbo Zhao2, Tagyoung Chung2, Jing Huang2, Nanyun Peng1,2 1 University of California, Los Angeles, 2 Amazon Alexa AI {yufeit,violetpeng}@cs.ucla.edu {naraanja,orabys,cervon,gsig,chenyt}@amazon.com {wenbzhao,tagyoung,jhuangz}@amazon.com Abstract Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical in- terest and more challenging than unconstrained lyric generation as the music imposes addi- tional constraints onto the lyrics. The train- ing data is limited as most songs are copy- righted, resulting in models that underfit the complicated cross-modal relationship between melody and lyrics. In this work, we propose a method for generating high-quality lyrics with- out training on any aligned melody-lyric data. Specifically, we design a hierarchical lyric gen- eration framework that first generates a song outline and second the complete lyrics. The framework enables disentanglement of training (based purely on text) from inference (melody- guided text generation) to circumvent the short- age of parallel data. We leverage the segmentation and rhythm align- ment between melody and lyrics to compile the given melody into decoding constraints as guidance during inference. The two-step hi- erarchical design also enables content control via the lyric outline, a much-desired feature for democratizing collaborative song creation. Ex- perimental results show that our model can gen- erate high-quality lyrics that are more on-topic, singable, intelligible, and coherent than strong baselines, for example SongMASS (Sheng et al., 2021), a SOTA model trained on a paral- lel dataset, with a 24% relative overall quality improvement based on human ratings. 1 1 Introduction Music is ubiquitous and an indispensable part of humanity (Edensor, 2020). Self-serve songwrit- ing has thus become an emerging task and has re- ceived interest by the AI community (Sheng et al., ∗Work was done when the author interned at Amazon. 1Our code is available at https:// github.com/amazon-science/ unsupervised-melody-to-lyrics-generation. Melody L S L S L S L S L Ma- ny skies ’ve turned to grey be- cause Hey, ba- by what ain’t nothing wrong to hide away So- me one got to - go - - Take me back to old- er time in lit- … Music Note Human Baseline 1 Baseline 2 Lyra (Ours) Figure 1: An example of the melody and the corre- sponding lyrics, where ‘L’ denotes a music note with long duration and ‘S’ stands for short. Our model LYRA generates more coherently than the baselines. Besides, the rhythms of lyrics (i.e., accents and relaxations when spoken) generated by human and LYRA align well with the flows of the melody. On the other hand, existing methods output lyrics that have low singability by either aligning multiple words with one single note (baseline 1) or vice versa (baseline 2) as highlighted in red. 2021; Tan and Li, 2021; Zhang et al., 2022; Guo et al., …

Content truncated to 500 words