Augmented Prompt Selection for Evaluation of Spontaneous Speech Synthesis

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6368–6374 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6368 Augmented Prompt Selection for Evaluation of Spontaneous Speech Synthesis ´Eva Sz´ekely, Jens Edlund, Joakim Gustafson Division of Speech, Music and Hearing, KTH Royal Institute of Technology Lindstedtsv¨agen 24, 114 28 Stockholm, Sweden szekely@kth.se, edlund@speech.kth.se, jkgu@kth.se Abstract By definition, spontaneous speech is unscripted and created on the fly by the speaker. It is dramatically different from read speech, where the words are authored as text before they are spoken. Spontaneous speech is emergent and transient, whereas text read out loud is pre-planned. For this reason, it is unsuitable to evaluate the usability and appropriateness of spontaneous speech synthesis by having it read out written texts sampled from for example newspapers or books. Instead, we need to use transcriptions of speech as the target - something that is much less readily available. In this paper, we introduce Starmap, a tool allowing developers to select a varied, representative set of utterances from a spoken genre, to be used for evaluation of TTS for a given domain. The selection can be done from any speech recording, without the need for transcription. The tool uses interactive visualisation of prosodic features with t-SNE, along with a tree-based algorithm to guide the user through thousands of utterances and ensure coverage of a variety of prompts. A listening test has shown that with a selection of genre-specific utterances, it is possible to show significant differences across genres between two synthetic voices built from spontaneous speech. Keywords: evaluation, spontaneous speech synthesis, human-in-the-loop, intelligence augmentation 1. Introduction The application areas of speech synthesis are expanding into new domains, from news readers and simple task- oriented dialogues to more elaborate, multi-turn conversa- tions. With this come new needs for more complex af- fordances such as exhibiting turn-taking cues, performing speaker entrainment, holding the listener’s attention or ex- pressing degrees of uncertainty, to name a few. It is ex- pected that agents and dialogue systems will move from using TTS trained on read-speech data to context embed- ded speech synthesisers, trained on recordings of real-life spontaneous speech. As we move away from citation-style, one-size-fits-all, neutral read speech, we find that more expressive TTS voices might also become more context- dependent (Sz´ekely et al., 2019b). For instance, a spon- taneous speech synthesiser trained on impromptu, casual conversations might perform really well on tasks within the original domain, but it is unknown if and how well it will transfer to new domains. However, it should not be nec- essary to record new data and train a new voice for each new application to ensure the appropriateness of the syn- thesiser’s speaking style. Rather, TTS developers should be equipped with rapid evaluation techniques, to measure a system’s performance for a particular domain, or compare systems across different genres and domains, and conse- quently adjust their systems for better suitability. When preparing materials for listening tests, evidently, the …

Content truncated to 500 words