Development of an Annotated Multimodal Dataset for the Investigation of Classification and Summarisation of Presentations using High-Level Paralinguistic Features

Development of an Annotated Multimodal Dataset for the Investigation of Classification and Summarisation of Presentations using High-Level Paralinguistic Features Keith Curtis⋆, Nick Campbell†, Gareth J. F. Jones⋆, ADAPT Centre⋆† School of Computing⋆, School of Computer Science & Statistics† Dublin City University⋆, Trinity College Dublin† Dublin, Ireland Keith.Curtis@dcu.ie, nick@tcd.ie, Gareth.Jones@dcu.ie Abstract Expanding online archives of presentation recordings provide potentially valuable resources for learning and research. However, the huge volume of data that is becoming available means that users have difficulty locating material which will be of most value to them. Conventional summarisation methods making use of text-based features derived from transcripts of spoken material can provide mechanisms to rapidly locate topically interesting material by reducing the amount of material that must be auditioned. However, these text-based methods take no account of the multimodal high-level paralinguistic features which form part of an audio-visual presentation, and can provide valuable indicators of the most interesting material within a presentation. We describe the development of a multimodal video dataset, recorded at an international conference, designed to support the exploration of automatic extraction of paralinguistic features and summarisation based on these features. The dataset is comprised of parallel recordings of the presenter and the audience for 31 conference presentations. We describe the process of performing manual annotation of high-level paralinguistic features for speaker ratings, audience engagement, speaker emphasis, and audience comprehension of these recordings. Used in combination these annotations enable research into the automatic classification of high-level paralinguistic features and their use in video summarisation. Keywords: Data Collection, Annotation, Classification 1. Introduction Online archives of presentations provide valuable sources of material for study and research. However, such is the amount of this content available in many settings it can be difficult for users to efficiently access material which is most likely to be of interest to them. Search of this con- tent based on the words spoken enables potentially relevant material to be identified based on the topic being presented. However, text-based analysis does not support location of the most important or emphasised material on a particular topic as indicated by the behaviour of the speaker or the audience. Use of high-level paralinguistic features which form part of the presentation offers the potential to iden- tify the most significant topically relevant material within a presentation. Research into the automatic identification of such paralinguistic features and their utilisation in guiding users to relevant material by the use of applications such as summarisation requires the development of suitable cor- pora to support this work. Such a dataset must include the relevant audio visual con- tent with suitable manual annotations of the features to be extracted. We describe the construction of such a corpus designed to support this research. Our corpus consists of recordings of paper presentations at the Speech Prosody 7 conference, an academic conference held in Dublin, Ireland in May 2014. The recorded contents include audio-visual content of the presenter, but also parallel recordings of the audience to each presentation. To investigate the automatic identification of paralinguis- …

Content truncated to 500 words