Evaluating Sampling-based Filler Insertion with Spontaneous TTS

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1960–1969 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1960 Evaluating Sampling-based Filler Insertion with Spontaneous TTS Siyang Wang, Joakim Gustafson, ´Eva Sz´ekely Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden {siyangw, jkgu, szekely}@kth.se Abstract Inserting fillers (such as “um”, “like”) to clean speech text has a rich history of study. One major application is to make dialogue systems sound more spontaneous. The ambiguity of filler occurrence and inter-speaker difference make both modeling and evaluation difficult. In this paper, we study sampling-based filler insertion, a simple yet unexplored approach to inserting fillers. We propose an objective score called Filler Perplexity (FPP). We build three models trained on two single-speaker spontaneous corpora, and evaluate them with FPP and perceptual tests. We implement two innovations in perceptual tests, (1) evaluating filler insertion on dialogue systems output, (2) synthesizing speech with neural spontaneous TTS engines. FPP proves to be useful in analysis but does not correlate well with perceptual MOS. Perceptual results show little difference between compared filler insertion models including with ground-truth, which may be due to the ambiguity of what is good filler insertion and a strong neural spontaneous TTS that produces natural speech irrespective of input. Results also show preference for filler-inserted speech synthesized with spontaneous TTS. The same test using TTS based on read speech obtains the opposite results, which shows the importance of using spontaneous TTS in evaluating filler insertions. Audio samples: www.speech.kth.se/tts-demos/LREC22 Keywords: filler insertion, spontaneous text-to-speech, spoken dialogue system 1. Introduction Spontaneous TTS has improved much recently due to usage of neural TTS engines. We can now gener- ate conversational speech phenomena such as hesita- tions, disfluencies (Sz´ekely et al., 2019b; Sz´ekely et al., 2019a) and breathing (Sz´ekely et al., 2020) in such TTS alone. A major hurdle in applying such TTS to a spoken dialogue system is that dialogue systems are mainly trained on written texts, which do not con- tain fillers such as “um” and “like”, that are common in the speech corpora spontaneous TTS systems are trained on (Sz´ekely et al., 2019a). This means that these systems can not take advantage of the potential of latest spontaneous TTS. Fillers are important for conversational systems, apart from making the speech sound more spontaneous, they can be used to commu- nicate planning problems (Levelt, 1983), handle turn- taking (Maclay and Osgood, 1959) and communicate personality (Gustafson et al., 2021). A recent speech- to-speech chat-bot system was trained on 2000 hours of transcribed telephone conversations (Nguyen et al., 2022). The speech part of the corpus was enough to reproduce non-verbal vocalizations, but the amount of text was not enough to achieve semantic consistency in the dialogue. Another solution would be to insert fillers into responses from a chat-bot system trained on a large text-based dialogue corpus. However, filler in- sertion is a difficult problem due to the ambiguity of fillers in natural speech …

Content truncated to 500 words