LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2750–2758 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2750 LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild David Gimeno-G´omez, Carlos-D. Mart´ınez-Hinarejos Pattern Recognition and Human Language Technologies Research Center, Universitat Polit`ecnica de Val`encia, Camino de Vera, s/n, 46022, Val`encia, Spain {dagigo1, cmartine}@dsic.upv.es Abstract Speech is considered as a multi-modal process where hearing and vision are two fundamentals pillars. In fact, several studies have demonstrated that the robustness of Automatic Speech Recognition systems can be improved when audio and visual cues are combined to represent the nature of speech. In addition, Visual Speech Recognition, an open research problem whose purpose is to interpret speech by reading the lips of the speaker, has been a focus of interest in the last decades. Nevertheless, in order to estimate these systems in the currently Deep Learning era, large-scale databases are required. On the other hand, while most of these databases are dedicated to English, other languages lack sufficient resources. Thus, this paper presents a semi-automatically annotated audiovisual database to deal with unconstrained natural Spanish, providing 13 hours of data extracted from Spanish television. Furthermore, baseline results for both speaker-dependent and speaker-independent scenarios are reported using Hidden Markov Models, a traditional paradigm that has been widely used in the field of Speech Technologies. Keywords: Audiovisual Database, Speech Recognition, Lipreading, Computer Vision 1. Introduction Despite the fact that speech perception is commonly considered as a purely auditory process, the truth is that it is a process involving multiple senses, as well as high-level knowledge related with grammar and se- mantics (Dupont and Luettin, 2000). In fact, the study carried out by McGurk and MacDonald (1976) demon- strated the importance of visual information and its re- lationship with the sounds produced. Nevertheless, for deaf or hearing-impaired people, who are totally or par- tially dependent on their sense of sight, speech under- standing poses a great challenge since, as Duchnowski et al. (2000) support, only 30% of speech informa- tion is visible. For this reason, different areas of re- search have focused their efforts on speech recogni- tion when the auditory sense is not functional, such as lipreading (Kaplan et al., 1987; Rodr´ıguez Ortiz, 2008), cued speech (Cornett, 1967), or silent speech interfaces (Denby et al., 2010). Regarding the field of Speech Technologies in its ori- gins, Automatic Speech Recognition (ASR) systems were focused only on processing the acoustic signal. Nowadays, this type of systems reaches high-quality performances (Chan et al., 2016). However, these ap- proaches suffer a deterioration in quality when the au- dio signal is damaged or corrupted (Juang, 1991). As a consequence, in order to deal with this issue, the research was impulsed towards Audio-Visual Speech Recognition (AVSR) approaches (Potamianos et al., 2003; Dupont and Luettin, 2000). In this way, as the above mentioned studies demonstrate, it was shown that the combination of acoustic and visual cues could represent the nature of speech more …

Content truncated to 500 words