Using the LARA Little Prince to compare human and TTS audio quality

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2967–2975 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2967 Using the LARA Little Prince to compare human and TTS audio quality Elham Akhlaghi1, Ingibjörg Iða Auðunardóttir2, Anna B ˛aczkowska3, Branislav Bédi4, Hakeem Beedar5, Harald Berthelsen6, Cathy Chua7, Catia Cucchiarini8, Hanieh Habibi9, Ivana Horváthová10, Junta Ikeda7, Christèle Maizonniaux11, Neasa Ní Chiaráin6, Chadi Raheb12, Manny Rayner9, John Sloan6,9, Nikos Tsourakis9, Chunlin Yao13 1Ferdowsi University of Mashhad, Iran; 2University of Iceland, Iceland; 3University of Gdansk, Gdansk, Poland; 4The Árni Magnússon Institute for Icelandic Studies, Iceland; 5University of Adelaide, Australia;6Trinity College, Dublin, Ireland; 7Independent scholar; 8Centre for Language and Speech Technology (CLST), Radboud University Nijmegen, The Netherlands; 9FTI/TIM, University of Geneva, Switzerland; 10Constantine the Philosopher University, Nitra, Slovakia; 11Flinders University, Adelaide, Australia; 12University of Guilan, Rasht, Iran 13Tianjin Chengjian University, Tianjin, China elhamakhlaghi80@gmail.com, iia2@hi.is, anna.baczkowska@ug.edu.pl, branislav.bedi@arnastofnun.is, hbeedar@hotmail.com.au, berthelh@tcd.ie, cathyc@pioneerbooks.com.au, c.cucchiarini@let.ru.nl, hanieh.habibi@unige.ch, ihorvathova@ukf.sk, ikedaj_91@hotmail.com, christele.maizonniaux@flinders.edu.au, Neasa.NiChiarain@tcd.ie,chadi.raheb@gmail.com, Emmanuel.Rayner@unige.ch, john.sloan.1@ucdconnect.ie, Nikolaos.Tsourakis@unige.ch, yao_chunlin@126.com Abstract A popular idea in Computer Assisted Language Learning (CALL) is to use multimodal annotated texts, with annotations typically including embedded audio and translations, to support second and foreign (L2) learning through reading. An important question is how to create good quality audio, which can be done either through human recording or by a Text-To-Speech (TTS) engine. We may reasonably expect TTS to be quicker and easier, but human to be of higher quality. Here, we report a study using the open source LARA platform and ten languages. Samples of audio totalling about five minutes, representing the same four passages taken from LARA versions of Saint-Exupèry’s Le petit prince, were provided for each language in both human and TTS form; the passages were chosen to instantiate the 2×2 cross product of the conditions {dialogue, not-dialogue} and {humour, not-humour}. 251 subjects used a web form to compare human and TTS versions of each item and rate the voices as a whole. For the three languages where TTS did best, English, French and Irish, the evidence from this study and the previous one it extended suggest that TTS audio is now pedagogically adequate and roughly comparable with a non-professional human voice in terms of exemplifying correct pronunciation and prosody. It was however still judged substantially less natural and less pleasant to listen to. No clear evidence was found to support the hypothesis that dialogue and humour pose special problems for TTS. All data and software will be made freely available. Keywords: TTS, evaluation, multimodality, reading, emotion 1. Introduction and overview In this paper, our central goal is to investigate the feasi- bility of using modern TTS technology in CALL applica- tions. The essential parameters of the problem are straight- forward. People who are trying to improve their abilities in L2 languages using a CALL application will often need L2 audio to train their listening skills. This audio can be provided by human voices or TTS engines. TTS makes it possible to produce audio quickly and cheaply and also …

Content truncated to 500 words