A Multimodal Educational Corpus of Oral Courses: Annotation, Analysis and Case Study

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4293–4301 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4293 A Multimodal Educational Corpus of Oral Courses: Annotation, Analysis and Case Study Salima Mdhaffar1,2, Yannick Est`eve2, Antoine Laurent1, Nicolas Hernandez3, Richard Dufour2, Delphine Charlet4, G´eraldine Damnati4, Solen Quiniou3, Nathalie Camelin2 1 LIUM - University of Le Mans, France 2 LIA - University of Avignon, France 3 LS2N - University of Nantes, France 4 Orange, France {firstname.lastname}@univ-lemans.fr,univ-avignon.fr,univ-nantes.fr,orange.com Abstract This paper presents a French multimodal educational dataset of oral courses. This corpus is part of the PASTEL (Performing Automated Speech Transcription for Enhancing Learning) project aiming to explore the potential of synchronous speech transcription and application in specific teaching situations (Bettenfeld et al., 2018; Bettenfeld et al., 2019). It includes 10 hours of different lectures, manually transcribed and segmented. The main interest of this corpus lies in its multimodal aspect: in addition to speech, the courses were filmed and the written presentation supports (slides) are made available. The dataset may then serve researches in multiple fields, from speech and language to image and video processing. The dataset will be freely available to the research community. In this paper, we first describe in details the annotation protocol, including a detailed analysis of the manually labeled data. Then, we propose some possible use cases of the corpus with baseline results. The use cases concern scientific fields from both speech and text processing, with language model adaptation, thematic segmentation and transcription to slide alignment. Keywords: Multimodal corpus, Educational context, Thematic segmentation, Alignment, Language model adaptation 1. Introduction With the increasing number of applications handling spon- taneous speech, lecture processing has becoming an active field of research. In this particular educational context, a large number of projects have been developed, coming with different datasets. Among them, we can first quote the LECTRA corpus (Trancoso et al., 2006; Trancoso et al., 2008) dealing with classroom lectures in European Por- tuguese. This corpus provides the audio of lectures and their manual transcription. In addition to the oral modality, the “Spontaneous Speech Corpus and Processing Technol- ogy”(Furui et al., 2000; Furui et al., 2001) and the CHIL projects (Lamel et al., 2005) include, in addition to the audio and the transcription, the video recording of lec- tures. Finally, the LMELectures corpus (Riedhammer et al., 2013) is the most complete one with various modalities (audio, video and text), including the annotation of speech transcription, a segmentation in speaker turn, as well as keywords definition. We propose in this paper an original French speech edu- cational corpus that includes 3 basic modalities: the oral lecture (audio), video recording and the presentation sup- ports (text). To these modalities are included additional manual annotations: manual transcriptions of lectures, in- domain words extraction and annotation, and alignment of presentation supports (slides) and oral presentation during the lectures. To our knowledge, there is no existing corpus that integrates such a variety of annotations. This paper aims at …

Content truncated to 500 words