Evaluation of TTS Systems in Intelligibility and Comprehension Tasks

Evaluation of TTS Systems in Intelligibility and Comprehension Tasks1 張瑜芸 Yu-Yun Chang 國立臺灣大學語言學研究所 Graduate Institute of Linguistics National Taiwan University june06029@gmail.com Abstract This paper aims at finding the relationships between intelligibility and comprehensibility in speech synthesizers, and tries to design an appropriate comprehension task for evaluating the speech synthesizers’ comprehensibility. It is predicted that speech synthesizer with higher intelligibility, will have greater performance in comprehension. Also, since the two most popular used speech synthesis methods are HMM-based and unit selection, this study tries to compare whether the HTS-2008 (HMM-based) or Multisyn (unit selection) speech synthesizer has better performance in application. Natural speech is applied in the experiment as a controlled group to the speech synthesizers. The results in the intelligibility test shows that natural speech is better than HTS-2008, and HTS-2008 is much better than Multisyn system. Whereas, in the comprehension task, all the three speech systems present not much differences in speech comprehending process. This is because that the two speech synthesizers have reached the threshold of enough intelligibility to provide high speech comprehension quality. Therefore, although with equal comprehensible speech quality between HTS-2008 and Multisyn systems, HTS-2008 speech synthesizer is more recommended and preferable due to its higher intelligibility. Keywords: speech synthesizers, intelligibility evaluation, comprehension evaluation, HTS-2008, Multisyn 1. Introduction Recently, text-to-speech (TTS) system synthesizers have been evaluated from different aspects, such as intelligibility, naturalness, and preference of the synthetic speech, as noted by [1]. Since the final purpose of applying the synthetic speech is to make it usable to applications, it is worth carrying out experiments measuring the synthesizers’ performance with human listeners. For measuring speech synthesizers, it was necessary to involve perception factors in synthetic speech evaluation, rather than merely evaluating the intelligibility, in order to better assess the speech synthesizers, as indicated by [2]. [3] also evaluated the aspect of the listener’s perception on a comprehension task to learn how well the synthetic speech was created by the synthesizers could be understood by the listeners. Moreover, [2] had demonstrated that there was a strong relationship between the intelligibility and comprehension. Also, they had specified the intelligibility was one of the important factors that would affect listening comprehension. It is then worth observing the relationships between intelligibility and comprehension for speech synthesizers. Although several studies have been successfully evaluating the intelligibility of speech synthesizers, very few researchers have examined the association with comprehension. However, it is hard 1 Acknowledgement: Thanks the people from University of Edinburgh - for the suggestions and comments offered by Simon King, and Catherine Mayo; the perception lab reserving and statistics helping supported by Rob Clark; and the HTS-2008 speech synthesizer provided by Junichi Yamagishi. 64 to measure comprehension, due to the fact that it involves cognitive processes which are hard to be captured and taken into account. Recent studies use post-perceptual comprehension tests to measure listeners’ comprehension, but many have failed to distinguish differences between TTS systems. An appropriate strategy for evaluating the comprehension is still not found. Therefore, this research …

Content truncated to 500 words