Learning the Beauty in Songs: Neural Singing Voice Beautifier

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7970 - 7983 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning the Beauty in Songs: Neural Singing Voice Beautifier Jinglin Liu and Chengxi Li and Yi Ren and Zhiying Zhu and Zhou Zhao∗ {jinglinliu,chengxili,rayeren,zhyingzh,zhaozhou}@zju.edu.cn Zhejiang University Abstract We are interested in a novel task, singing voice beautification (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are im- mature, and most of them are restricted to in- tonation but ignore the overall aesthetic qual- ity. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping ap- proach for pitch correction: Shape-Aware Dy- namic Time Warping (SADTW), which ame- liorates the robustness of existing time-warping approaches, to synchronize the amateur record- ing with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. To achieve this, we also propose a new dataset containing parallel singing recordings of both amateur and profes- sional versions. Extensive experiments on both Chinese and English songs demonstrate the ef- fectiveness of our methods in terms of both objective and subjective metrics. Audio sam- ples are available at https://neuralsvb. github.io. Codes: https://github. com/MoonInTheRiver/NeuralSVB. 1 Introduction The major successes of the artificial intelligent singing voice research are primarily in Singing Voice Synthesis (SVS) (Lee et al., 2019; Blaauw and Bonada, 2020; Ren et al., 2020; Lu et al., 2020; Liu et al., 2021a) and Singing Voice Conversion (SVC) (Sisman and Li, 2020; Li et al., 2021; Wang et al., 2021a). However, the Singing Voice Beautifi- cation (SVB) remains an important and challenging endeavor for researchers. SVB aims to improve the ∗ Corresponding Author intonation1 and the vocal tone of the voice, while keeping the content and vocal timbre2. SVB is ex- tensively required both in the professional record- ing studios and the entertainment industries in our daily life, since it is impractical to record flawless singing audio. Nowadays in real-life scenarios, SVB is usually performed by professional sound engineers with adequate domain knowledge, who manipulate com- mercial vocal correction tools such as Melodyne3 and Autotune4 (Yong and Nam, 2018). Most cur- rent automatic pitch correction works are shown to be an attractive alternative, but they may 1) show weak alignment accuracy (Luo et al., 2018) or pitch accuracy (Wager et al., 2020); 2) cause the tuned recording and the reference recording to be homo- geneous in singing style (Yong and Nam, 2018). Besides, they typically focus on the intonation but ignore the overall aesthetic quality (audio quality and vocal tone) (Rosenzweig et al., 2021; Zhuang et al., 2021). To tackle these …

Content truncated to 500 words