Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel’s Weekly Video Podcasts

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2536–2540 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2536 Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel’s Weekly Video Podcasts Debjoy Saha1,2, Shravan Nayak1,3, Timo Baumann1,4 1Universit¨at Hamburg, 2IIT Kharagpur, 3IIT (BHU) Varanasi, 4OTH Regensburg sahadebjoy10@iitkgp.ac.in, pshravan.nayak.ece17@itbhu.ac.in, timo.baumann@oth-regensburg.de Abstract We introduce the Merkel Podcast Corpus, an audio-visual-text corpus in German collected from 16 years of (almost) weekly Internet podcasts of former German chancellor Angela Merkel. To the best of our knowledge, this is the first single speaker corpus in the German language consisting of audio, visual and text modalities of comparable size and temporal extent. We describe the methods used with which we have collected and edited the data which involves downloading the videos, transcripts and other metadata, forced alignment, performing active speaker recognition and face detection to finally curate the single speaker dataset consisting of utterances spoken by Angela Merkel. The proposed pipeline is general and can be used to curate other datasets of similar nature, such as talk show contents. Through various statistical analyses and applications of the dataset in talking face generation and TTS, we show the utility of the dataset. We argue that it is a valuable contribution to the research community, in particular, due to its realistic and challenging material at the boundary between prepared and spontaneous speech. Keywords: multi-modal, German, single-speaker, corpus, speaker diarization, forced alignment, cross-modal learning Figure 1: Angela Merkel looking in all directions during video podcasts. 1. Introduction Multi-modal spoken video corpora are becoming more widespread and relevant in recent years. Given today’s compute power, multi-modal analyses as well as cross- modal learning between video and audio – and par- ticularly: between facial mimicry and speech – have become more commonplace. For example, based on weekly addresses of then-president of the United States, Barack Obama, ObamaNet (Kumar et al., 2018) pro- duces speech and lip-synchronized facial videos from text and some intermediate representation. There are a number of spoken video corpora available for end-to-end audio-visual machine learning tasks. The LRS 2 (Chung et al., 2016) and LRS 3 (Afouras et al., 2018) datasets contain thousands of sentence-like units from many different speakers collected from the BBC and from TED talks, i. e. spoken in the wild. The broadness and wide coverage of such multi-spea- ker corpora come with some limitations. In particular, speech samples are devoid of their original context as short snippets are available only in isolation. Very lit- tle material is available per-speaker and snippets tend to be short and typically contain only single phrases. This prohibits modelling long-range phenomena such as conversational prosody and the focus on short snippets makes it impossible to model speaker behaviour during short speech pauses. On the other hand, there is also already a number of single-speaker video corpora collected from speeches of public figures, such as former US presidents Obama (Janssoone et al., 2020) and Trump (Ruf and …

Content truncated to 500 words