The Badalona Corpus - An Audio, Video and Neuro-Physiological Conversational Dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5170–5177 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5170 The Badalona Corpus An Audio, Video and Neuro-Physiological Conversational Dataset Philippe Blache1, Salom´e Antoine2, Dorina De Jong3, Lena-Marie Huttner1, Emilia Kerr1, Thierry Legou1, Eliot Ma¨es4, Cl´ement Franc¸ois1 1Laboratoire Parole & Langage (CNRS-Aix-Marseille Univ.), 2Freie Universit¨at Berlin, 3Center for Translational Neurophysiology of Speech and Communication (Universit`a di Ferrara), 4Laboratoire Informatique & Syst`emes (CNRS-Aix-Marseille Univ.) blache@ilcb.fr, salome.antoine@fu-berlin.de, dorina.dejong@iit.it, lena-marie.huttner@univ-amu.fr, emilia.kerr@univ-amu.fr, eliot.maes@etu.univ-amu.fr, thierry.legou@univ-amu.fr, clement.francois@univ-amu.fr Abstract We present in this paper the first natural conversational corpus recorded with all modalities and neuro-physiological signals. Five dyads (10 participants, Spanish native speakers) have been recorded three times, during three sessions (about 30 minutes each) with 4 days interval. During each session, audio and video are captured as well as the neural signal (EEG with Emotiv-EPOC) and the electro-physiological one (with Empatica-E4). This resource is original in several respects. Technically, it is the first one gathering all these types of data in a natural conversation situation. Moreover, the recording of the same dyads at different periods opens the door to new longitudinal investigations such as the evolution of interlocutors’ alignment over time. The paper situates this new type of resources in the literature, presents the experimental setup and describes different annotations enriching the corpus. Keywords: Multimodal corpus, natural conversation, convergence, neuro-physiological data, EEG 1. Introduction: studying brain basis of language processing in a natural context Studying language processing in a natural context means that various sources of information need to be taken into account. Many studies have been devoted to the interaction between the different modalities, ver- bal and non-verbal (Jewitt, 2013). They have explored in detail how the different sources of information in- teract in order to encode, transmit and decode infor- mation during a natural interaction between interlocu- tors (Pickering and Garrod, 2021). This new perspec- tive has laid the ground for investigation of the under- pinning mechanisms of natural interaction by analyz- ing the neuro-physiological signals elicited by this phe- nomenon. The aim of the present project is to study, on the one hand, the physiological signals (electro-dermal reaction, temperature, heart rate, breathing) and, on the other hand, the neural correlates (in particular, neural oscillations at different frequency bands) of language processing. The goal when addressing this type of question is to look for correlates between the neuro- physiological signal and typical phenomena of human interaction such as emotion, engagement in the dia- logue, convergence effects, information exchange, etc. Research in this direction has been quite limited so far for various reasons. In particular, at the theoretical level, we still do not have a global view on how the dif- ferent sources of information interact during a conver- sation in order to build and exchange meaning. More- over, we need to explore these aspects both at a local level (i.e., a specific moment of the interaction) and the global one (i.e., the entire interaction). However, at …

Content truncated to 500 words