A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6829–6836 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6829 A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning Gerald Schwiebert, Cornelius Weber, Leyuan Qu, Henrique Siqueira, Stefan Wermter Knowledge Technology, Department of Informatics, University of Hamburg Vogt-Koelln-Str. 30, 22527 Hamburg g.schwiebert@mailbox.org {weber, qu, siqueira, wermter}@informatik.uni-hamburg.de Abstract Large datasets as required for deep learning of lip reading do not exist in many languages. In this paper we present the dataset GLips (German Lips) consisting of 250,000 publicly available videos of the faces of speakers of the Hessian Parliament, which was processed for word-level lip reading using an automatic pipeline. The format is similar to that of the English language LRW (Lip Reading in the Wild) dataset, with each video encoding one word of interest in a context of 1.16 seconds duration, which yields compatibility for studying transfer learning between both datasets. By training a deep neural network, we investigate whether lip reading has language-independent features, so that datasets of different languages can be used to improve lip reading models. We demonstrate learning from scratch and show that transfer learning from LRW to GLips and vice versa improves learning speed and performance, in particular for the validation set. Keywords: Audio-visual, Dataset, Lip reading, Automatic Speech Recognition, Deep Learning, Transfer Learning, Computer Vision 1. Introduction Lip reading is the ability of drawing conclusions about what is being said by visually observing a speaker’s lips. In practice, however, it can rarely be considered on its own in human speech, since a variety of addi- tional information is usually available in communica- tion which, in combination with lip reading, can in- crease the information content of the incoming com- munication (Klucharev et al., 2003). These can con- sist of, for example, audio information, context, heuris- tics, gestures, facial expressions, or prior knowledge about what is being said. Also, special capabilities of the human brain and auditory sense allow us to ap- ply filters that increase the focus on the desired com- munication, such as the well-known cocktail party ef- fect (Cherry, 1953). Thus, when phonemes, syllables, words, or entire sentences have an information deficit from the sender or receiver related to the communica- tion pathway, lip reading can be one among many ways to compensate for this deficit. Ambient noise, hearing loss, slurred pronunciation, unfamiliar words, distance, or soundproof barriers are examples of communication problems between sender and receiver. Most people automatically look at the speaker’s lips when intelligi- bility suffers, so they are all lip readers with varying degrees of skill (Woodhouse et al., 2009). There is a variety of possible applications for lip read- ing, such as disability support, sports communication in the press, voice control in noisy environments, ad- ditional accuracy for ASR systems, law enforcement and preservation of evidence. In recent years, enor- mous progress has been made in a number of techni- cal areas, the effects of which …

Content truncated to 500 words