RoomReader: A Multimodal Corpus of Online Multiparty Conversational Interactions

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2517–2527 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2517 RoomReader: A Multimodal Corpus of Online Multiparty Conversational Interactions Justine Reverdy,∗† Sam O’Connor Russell,∗† Louise Duquenne,∗ Diego Garaialde,‡ Benjamin Cowan,‡ Naomi Harte∗† ∗ Sigmedia Group, ADAPT Centre, School of Engineering, Trinity College Dublin ‡ADAPT Centre, School of Information and Communication Studies, University College Dublin †{reverdyj, russelsa, nharte}@tcd.ie Abstract We present RoomReader, a corpus of multimodal, multiparty conversational interactions in which participants followed a collaborative student-tutor scenario designed to elicit spontaneous speech. The corpus was developed within the wider RoomReader Project to explore multimodal cues of conversational engagement and behavioural aspects of collaborative interaction in online environments. However, the corpus can be used to study a wide range of phenomena in online multimodal interaction. The publicly-shared corpus consists of over 8 hours of video and audio recordings from 118 participants in 30 gender-balanced sessions, in the “in-the-wild” online environment of Zoom. The recordings have been edited, synchronised, and fully transcribed. Student participants have been continuously annotated for engagement with a novel continuous scale. We provide questionnaires measuring engagement and group cohesion collected from the annotators, tutors and participants themselves. We also make a range of accompanying data available such as personality tests and behavioural assessments. The dataset and accompanying psychometrics present a rich resource enabling the exploration of a range of downstream tasks across diverse fields including linguistics and artificial intelligence. This could include the automatic detection of student engagement, analysis of group interaction and collaboration in online conversation, and the analysis of conversational behaviours in an online setting. Keywords: multimodal corpus, multiparty interaction, online interaction, video-conferencing, engagement detection 1. Introduction To enable researchers to have a better understanding of complex communication behaviours that occur during social interactions, large labelled datasets of specific types of interactional situations are required (Ruhi et al., 2014). These datasets help uncover the links that tie high level phenomena such as conversational engage- ment, dominance or mutual understanding, and timed social signals contained in verbal and non-verbal be- haviours (Anderson et al., 1991; Janin et al., 2003; Mc- Cowan et al., 2005; McKeown et al., 2011; Ringeval et al., 2013; Gupta et al., 2016). These links are ex- plored to detect markers and find patterns that can be analysed and automatically extracted from features in audio and visual signals. Throughout 2020 and 2021, multiple lockdowns were enforced in many countries to slow the spread of the COVID-19 pandemic. Almost overnight online education became commonplace, pre- viously having been a relatively niche practice. Work- ers, teachers and students, had to rapidly adapt to an un- familiar paradigm to continue their professional, edu- cational and social activities. This situation confronted many with the range of issues that accompany video- conferencing interactions, e.g., fatigue (Fauville et al., 2021), time latency leading to missed or distorted turn- taking cues (Seuren et al., 2021), or difficulties to per- ceive disengagement cues from interlocutors …

Content truncated to 500 words