DNN-based Speech Synthesis Using Abundant Tags of Spontaneous Speech Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6438–6443 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6438 DNN-based Speech Synthesis Using Abundant Tags of Spontaneous Speech Corpus Yuki Yamashita1, Tomoki Koriyama2, Yuki Saito2, Shinnosuke Takamichi2, Yusuke Ijima3, Ryo Masumura3, Hiroshi Saruwatari2 1Faculty of Engineering, The University of Tokyo, 2Graduate School of Information Science and Technology, The University of Tokyo, 3NTT Media Intelligence Laboratories, NTT Corporation, 1,2 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan yukiyama913@g.ecc.u-tokyo.ac.jp, tomoki koriyama@ipc.i.u-tokyo.ac.jp Abstract In this paper, we investigate the effectiveness of using rich annotations in deep neural network (DNN)-based statistical speech synthesis. DNN-based frameworks typically use linguistic information as input features called context instead of directly using text. In such frameworks, we can synthesize not only reading-style speech but also speech with paralinguistic and nonlinguistic features by adding such information to the context. However, it is not clear what kind of information is crucial for reproducing paralinguistic and nonlinguistic features. Therefore, we investigate the effectiveness of rich tags in DNN-based speech synthesis according to the Corpus of Spontaneous Japanese (CSJ), which has a large amount of annotations on paralinguistic features such as prosody, disfluency, and morphological features. Experimental evaluation results shows that the reproducibility of paralinguistic features of synthetic speech was enhanced by adding such information as context. Keywords: speech synthesis, context, spontaneous speech, annotation, deep neural network 1. Introduction Speech synthesis is used in various applications such as smart speakers and robots, and situations such as announce- ments in public transportation and voice-overs of videos. A widely-studied speech synthesis is DNN-based one (Zen et al., 2013) in which the relationship between a text and an acoustic feature sequence is modeled by DNN. However, if we directly use text as an input of DNN, the training tends to be difficult. Instead, context is widely used as a model in- put. Context is defined as a variation factor of phonetic and prosodic features, and there are various contextual factors such as adjacent phones and stress in syllables. We encode contextual factors into a real-valued vector to enable DNN training. In this study, we focus on the construction of context. Al- though general reading-style speech synthesis only uses lin- guistic information as a context, context is more flexible, namely, it is easy to feed various information into the input of DNN. For example, inputting a speaker vector is suc- cessful in multi-speaker modeling (Wu et al., 2015; Hojo et al., 2018). The emotion control of synthetic speech was achieved by using the degree of expressiveness as a contex- tual factor (An et al., 2017; Lorenzo-Trueba et al., 2018). It is also reported that local features such as emphasis can be used as the input of DNN (Wang et al., 2018). In order to synthetize speech with a wide variety of paralin- guistic features, adding such features as context is promis- ing. However, too many input features often makes training difficult and causes overfitting problem. Therefore, in this paper, …

Content truncated to 500 words