Studying Common Ground Instantiation Using Audio, Video and Brain Behaviours: The BrainKT Corpus

Proceedings of Recent Advances in Natural Language Processing, pages 691–702 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_075 691 Studying Common Ground Instantiation Using Audio, Video and Brain Behaviours: The BrainKT Corpus Eliot Ma¨es Leonor Becerra-Bonache Aix-Marseille Universit´e, CNRS, LIS, Marseille, France eliot.maes@lis-lab.fr leonor.becerra@lis-lab.fr Thierry Legou Philippe Blache Aix-Marseille Universit´e, CNRS, LPL, Marseille, France thierry.legou@univ-amu.fr blache@ilcb.fr Abstract An increasing amount of multimodal record- ings has been paving the way for the develop- ment of a more automatic way to study lan- guage and conversational interactions. How- ever this data largely comprises of audio and video recordings, leaving aside other modali- ties that might complement this external view of the conversation but might be more diffi- cult to collect in naturalistic setups, such as participants brain activity. In this context, we present BrainKT, a natural conversational cor- pus with audio, video and neuro-physiological signals, collected with the aim of studying in- formation exchanges and common ground in- stantiation in conversation in a new, more in- depth way. We recorded conversations from 28 dyads (56 participants) during 30 minutes experiments where subjects were first tasked to collaborate on a joint information game, then freely drifted to the topic of their choice. Dur- ing each session, audio and video were cap- tured, along with the participants’ neural sig- nal (EEG with Biosemi 64) and their electro- physiological activity (with Empatica-E4). The paper situates this new type of resources in the literature, presents the experimental setup and describes the different kinds of annotations con- sidered for the corpus. 1 Introduction Language processing in a natural context is inher- ently multimodal, and many studies have been de- voted to better understanding how the interactions between the different channels leads to a better un- derstanding between participants of a conversation. Interaction theories (Pickering and Garrod, 2021) postulate that this understanding is based on an operation of information transfer between partici- pants, leading to the establishment of a common ground of knowledge. These processes happen at different levels, and the encoding and transmitting of information can be manifested through various cues for the different sources. These include feed- back from gestures, gaze and facial expressions (Bavelas et al., 2000) and are also manifested at the physiological level with variations in respiratory rate, heart rate, skin temperature, etc. (Włodarczak and Heldner, 2016). Less perceivable to the other speaker but not less interesting for the understand- ing of their behavior, the brain activity denote of specific rhythmic activity when alignment between speakers occurs in a conversation, with the 10- 12Hz (mu) frequency band presenting a specific pattern in the integration of mutual information during an interaction as well as in the coordination between speakers (Mandel et al., 2016; P´erez et al., 2017b; Menenti et al., 2012; Silbert et al., 2014). These new perspectives have laid the ground for investigations of natural conversations using neuro- physiological elicited; however enterprises into this domain remain few in number, for various reasons. The main constraint indeed remains the technical difficulty to create a corpus of natural …

Content truncated to 500 words