Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 544–551 November 10–11, 2021. ©2021 Association for Computational Linguistics 544 Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation Siddique Latif†∗ Inyoung Kim‡ Ioan Calapodescu‡ Laurent Besacier‡ †University of Southern Queensland, Australia ‡NAVER LABS Europe first.last@naverlabs.com Abstract While End-2-End Text-to-Speech (TTS) has made significant progresses over the past few years, these systems still lack intuitive user controls over prosody. For instance, generat- ing speech with fine-grained prosody control (prosodic prominence, contextually appropri- ate emotions) is still an open challenge. In this paper, we investigate whether we can control prosody directly from the input text, in order to code information related to contrastive focus which emphasizes a specific word that is con- trary to the presuppositions of the interlocutor. We build and share a specific dataset for this purpose and show that it allows to train a TTS system were this fine-grained prosodic feature can be correctly conveyed using control to- kens. Our evaluation compares synthetic and natural utterances and shows that prosodic pat- terns of contrastive focus (variations of Fo, In- tensity and Duration) can be learnt accurately. Such a milestone is important to allow, for ex- ample, smart speakers to be programmatically controlled in terms of output prosody. Index Terms: End-to-End TTS, fine-grained prosody control, contrastive focus, interrogative/assertive sen- tences. 1 Introduction Text-to-Speech (TTS) systems attempt to produce human-like speech by processing natural language text inputs. Neural network based TTS has made rapid progresses and attracted a strong attention in recent years (Wang et al., 2017; Shen et al., 2018b; Ła´ncucki, 2021; Valle et al., 2020). Not only synthetic speech quality but also inference speed were improved, the latter benefiting from non-autoregressive TTS models (Ren et al., 2019, 2020). Controlling prosody is an- other issue, as features such as pitch and duration are difficult to predict because of their large fluctuations over time. While recent works proposed to synthe- size speech which closely resembles the prosody of ∗This study was conducted during the internship at NAVER LABS Europe. a provided reference speech using latent representa- tions (Skerry-Ryan et al., 2018; Lee and Kim, 2019; Habib et al., 2019; Sun et al., 2020b), this work ad- dresses explicit user control of prosody using the sym- bolic (text) input. Our case study concerns control of contrastive focus which emphasizes on a specific word that is contrary to the presuppositions of the interlocu- tor (figure 1). More precisely, we want the TTS system to correctly convey the information initially transmit- ted by the speaker including prosodic prominence (for instance we want to emphasize the word HOUSE in the sentence ’Sarah closed the HOUSE’). We posit that controlling prosody in TTS is impor- tant for future prosody transfer in speech-to-speech translation systems (carrying the real meaning of a source utterance spoken by a human). For instance, spoken utterances of figure 1 in English (where focus is only prosodically marked) would lead to different word orderings if translated to …

Content truncated to 500 words