Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3886–3896 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3886 Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework ∗ Mingbo Ma†∗ Baigong Zheng†§∗ Kaibo Liu† Renjie Zheng† Hairong Liu† Kainan Peng†¶ Kenneth Church† Liang Huang†,‡ †Baidu Research, Sunnyvale, CA, USA ‡Oregon State University, Corvallis, OR, USA {mingboma, lianghuang}@baidu.com Abstract Text-to-speech synthesis (TTS) has witnessed rapid progress in recent years, where neural methods became capable of producing audios with high naturalness. However, these efforts still suffer from two types of latencies: (a) the computational latency (synthesizing time), which grows linearly with the sentence length, and (b) the input latency in scenarios where the input text is incrementally available (such as in simultaneous translation, dialog generation, and assistive technologies). To reduce these la- tencies, we propose a neural incremental TTS approach using the prefix-to-prefix framework from simultaneous translation. We synthesize speech in an online fashion, playing a segment of audio while generating the next, resulting in an O(1) rather than O(n) latency. Experi- ments on English and Chinese TTS show that our approach achieves similar speech natural- ness compared to full sentence TTS, but only with a constant (1–2 words) latency. 1 Introduction Text-to-speech synthesis (TTS) generates speech from text, and is an important task with wide ap- plications in dialog systems, speech translation, natural language user interface, assistive technolo- gies, etc. Recently, it has benefited greatly from deep learning, with neural TTS systems becoming capable of generating audios with high naturalness (Oord et al., 2016; Shen et al., 2018). State-of-the-art neural TTS systems generally consist of two stages: the text-to-spectrogram stage which generates an intermediate acoustic repre- sentation (linear- or mel-spectrogram) from the ∗M. M. and B. Z. contributed equally; M. M. co-directed the project (with L. H.), and was responsible for the majority of ideas and implementations; B. Z. improved the speech quality significantly and implemented the ideas on different TTS systems. See our generated audio samples and demos at https://inctts.github.io/. §Currently address: Kwai Inc., Seattle, WA. ¶Current address: Amazon, San Francisco, CA. “Start to record the voice.” Start to record the voice Start to record the voice Figure 1: Monotonic spectrogram-to-text attention. text, and the spectrogram-to-wave stage (vocoder) which converts the aforementioned acoustic repre- sentation into actual wave signals. In both stages, there are sequential approaches based on the seq- to-seq framework, as well as more recent parallel methods. The first stage, being relatively fast, is usually sequential (Wang et al., 2017; Shen et al., 2018; Li et al., 2019) with a few exceptions (Ren et al., 2019; Peng et al., 2019), while the second stage, being much slower, is more commonly par- allel (Oord et al., 2018; Prenger et al., 2019). Despite these successes, standard full-sentence neural TTS systems still suffer from two types of latencies: (a) the computational latency (synthesiz- ing time), which still grows linearly with the sen- tence length even using parallel inference (esp. in the second stage), and (b) the input latency in sce- …

Content truncated to 500 words