CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9317–9331 July 9-14, 2023 ©2023 Association for Computational Linguistics CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training Zhenhui Ye∗ zhenhuiye@zju.edu.cn Zhejiang University Rongjie Huang∗ rongjiehuang@zju.edu.cn Zhejiang University Yi Ren ren.yi@bytedance.com Bytedance Ziyue Jiang jiangziyue@zju.edu.cn Zhejiang University Jinglin Liu liu.jinglin@bytedance.com ByteDance Jinzheng He jinzhenghe@zju.edu.cn Zhejiang University Xiang Yin yixiang.stephen@bytedance.com Bytedance Zhou Zhao† zhaozhou@zju.edu.cn Zhejiang University Abstract Improving text representation has attracted much attention to achieve expressive text-to- speech (TTS). However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low train- ing efficiency and difficulty in prosody model- ing. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that explic- itly learns the prosody variance of the same text token under different contexts. Specifi- cally, 1) We encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space with the elaborate design of the encoder inputs and con- trastive loss; 2) We introduce a multi-scale pre- training pipeline to capture prosody patterns in multiple levels. We show how to incor- porate CLAPSpeech into existing TTS mod- els for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its gener- alization ability to adapt to multiple languages and multi-speaker TTS. We also deeply ana- lyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in our method. Source code and audio samples are available at https://clapspeech.github.io. 1 Introduction With the development of deep learning, the audio quality of modern TTS systems has been improved, yet prosody modeling is still a challenging problem. Previous works on expressive TTS have utilized external variation predictors (prediction-based, PB) ∗Equal contribution. †Corresponding author. (Ren et al., 2021a) and variational generative mod- els (variation-based, VB) (Kim et al., 2020; Liu et al., 2022) to inject prosody variance into the TTS model. Another popular direction is to learn better text representation for prosody prediction (Tan et al., 2021). However, the existing text rep- resentation learning methods for TTS are either based on the masked language model task (Devlin et al., 2019; Jia et al., 2021; Chen et al., 2021) (i.e., learn a BERT-like large language model on a text corpus) or masked acoustic model task (Chen et al., 2020; Bai et al., 2022) (i.e., reconstruct the masked mel-spectrogram based on the input text), which result in two disadvantages. Firstly, they only implicitly learn prosody with reconstruction losses, which distracts the model from improving the prosody modeling. Secondly, they do not de- couple the pronunciation space and prosody space, which leads to low training efficiency and a waste of model capacity. We perform a case study in Section 4.3.1, in which we can see that previous text representation used in TTS cannot capture the prosody variance under different text contexts. Technically, prosody can be regarded as the pitch and …

Content truncated to 500 words