Learning Robust Latent Representations for Controllable Speech Synthesis

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3562–3575 August 1–6, 2021. ©2021 Association for Computational Linguistics 3562 Learning Robust Latent Representations for Controllable Speech Synthesis Shakti Kumar*† Jithin Pradeep* Hussain Zaidi* *Vanguard Center for Analytics and Insights †Computer Science, University of Toronto {shakti kumar, jithin pradeep, hussain zaidi}@vanguard.com Abstract State-of-the-art Variational Auto-Encoders (VAEs) for learning disentangled latent representations give impressive results in discovering features like pitch, pause duration, and accent in speech data, leading to highly controllable text-to-speech (TTS) synthesis. However, these LSTM-based VAEs fail to learn latent clusters of speaker attributes when trained on limited or noisy datasets. Further, different latent variables are found to encode the same features, limiting the control and expressiveness during speech synthesis. To resolve these issues, we propose REMMI (Re- ordered transformer Encoder with Minimal Mutual Information) where we minimize the mutual information between different latent variables and devise a modified Transformer architecture with layer reordering to learn controllable latent representations in speech data. We show that REMMI reduces the cluster overlap of speaker attributes by at least 30% over LSTM-VAE. 1 Introduction Learning disentangled latent representations in speech is an active area of research (Hsu et al., 2017; Chou et al., 2018; Park et al., 2020) with applications in controlling the style (for example, pitch, pause duration, and accent) of synthesized speech. Recurrent architectures like Long Short Term Memory (LSTM) (Hochreiter and Schmidhu- ber, 1997) networks in Variational Autoencoders (VAE) have been state-of-the-art in discovering dis- entangled latent representations in speech (Wang et al., 2018; Jia et al., 2018; Skerry-Ryan et al., 2018) as well as sequential data more generally. For example Li and Mandt (2018) attempt to disen- tangle global and local features of video/speech in different latent variables. Hsu et al. (2019) disen- tangled different dimensions of the latent variables to discover meaningful representations and hence proposed a speech synthesis model with control- lable pitch, pause duration, and speed. These papers as well as several others (Chung et al., 2015; Hsu et al., 2019; Leglaive et al., 2020; Hono et al., 2020; Sun et al., 2020) make one lim- iting assumption— the availability of hundreds of hours of speech data for training deep learning networks. As we show in our experiments, state- of-the-art VAEs fail to learn meaningful separation of speaking styles in speech data when presented with small datasets. In addition, different latent variables learned by the VAE are no longer un- correlated. Both these shortcomings lead to poor control of speaking styles during synthesis. While LSTMs are state-of-the-art in learning latent variables in speech, Transformers have been used for understanding latent representations for text completion (Wang and Wan, 2019) and Transformer-based VAEs were used in Jiang et al. (2020) to model independent style attributes in mu- sic generation. Inspired by these limitations of LSTM-based VAEs and the promise of more ”attentive” net- works, we modify the loss function of the state- of-the-art VAEs (Hsu et al., 2019) by explicitly minimizing the mutual information between latent variables, …

Content truncated to 500 words