Unsupervised Multi-scale Expressive Speaking Style Modeling with Hierarchical Context Information for Audiobook Speech Synthesis

Proceedings of the 29th International Conference on Computational Linguistics, pages 7193–7202 October 12–17, 2022. 7193 Unsupervised Multi-scale Expressive Speaking Style Modeling with Hierarchical Context Information for Audiobook Speech Synthesis Xueyuan Chen∗ Shenzhen International Graduate School Tsinghua University Shenzhen, China chenxuey20@mails. tsinghua.edu.cn Shun Lei Shenzhen International Graduate School Tsinghua University Shenzhen, China leis21@mails. tsinghua.edu.cn Zhiyong Wu† Shenzhen International Graduate School Tsinghua University Shenzhen, China zywu@se.cuhk.edu.hk Dong Xu Tencent Music Entertainment Group Shenzhen, China huberyxu@tencent.com Weifeng Zhao Tencent Music Entertainment Group Shenzhen, China ethanzhao@tencent.com Helen Meng The Chinese University of Hong Kong Hong Kong SAR, China hmmeng@se.cuhk.edu.hk Abstract Naturalness and expressiveness are crucial for audiobook speech synthesis, but now are lim- ited by the averaged global-scale speaking style representation. In this paper, we propose an un- supervised multi-scale context-sensitive text-to- speech model for audiobooks. A multi-scale hi- erarchical context encoder is specially designed to predict both global-scale context style em- bedding and local-scale context style embed- ding from a wider context of input text in a hierarchical structure. Likewise, a multi-scale reference encoder is introduced to extract refer- ence style embeddings at both global and local scales from the reference speech, which are used to guide the prediction of speaking styles. On top of these, a bi-reference attention mech- anism is used to align both local-scale refer- ence style embedding sequence and local-scale context style embedding sequence with corre- sponding phoneme embedding sequence. Both objective and subjective experiment results on a real-world multi-speaker Mandarin novel audio dataset demonstrate the excellent performance of our proposed method over all baselines in terms of naturalness and expressiveness of the synthesized speech1. ∗ Work conducted when the first author was intern at Tencent Music Entertainment Group. † Corresponding author. 1Synthesized speech samples are available at: https://thuhcsi.github.io/COLING2022-MSHCE-TTS 1 Introduction Recently, some text-to-speech (TTS) models, such as Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018), Deep voice (Ping et al., 2017), Trans- formerTTS (Li et al., 2019) have been proposed to generate speech autoregressively from text input, and can achieve performance very close to human quality. In order to increase inference speed and generate more robust speech, non-autoregressive TTS models such as FastSpeech (Ren et al., 2019) and FastSpeech 2 (Ren et al., 2020) are emerged with robust and fast parallel generation. However, limited expressiveness of synthesized audio persists as one of the major gaps between synthesized speech and real human speech, which draws growing attention to expressive speech syn- thesis studies. Synthesizing long-form expressive datasets (such as audiobooks) is still a challeng- ing task, since wide-ranging voice characteristics are collapsed into an averaged prosodic style. To address this issue, style transfer TTS has been a popular strategy in recent years (Skerry-Ryan et al., 2018). The global style token (GST) model (Wang et al., 2018) adopts multi-head attention mecha- nism and several learnable style tokens to extract the global style from reference audio in an unsuper- vised way. Further more, a hierarchical GST archi- tecture is proposed to learn hierarchical embedding information implicitly, which contains …

Content truncated to 500 words