Multidimensional Coding of Multimodal Languaging in Multi-Party Settings

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2781–2787 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2781 Multidimensional Coding of Multimodal Languaging in Multi-Party Settings Christophe Parisse1, Marion Blondel2, Stéphanie Caët3, Claire Danet4, Coralie Vincent2, Aliyah Morgenstern5 1INSERM, Modyco, CNRS & Paris Nanterre University, Nanterre, France ; 2SFL, CNRS & Paris8, France 3STL, CNRS & Lille University, Lille, France 4Dylis, Normandie University, Rouen, France 5PRISMES, Sorbonne Nouvelle University, Paris, France cparisse@parisnanterre.fr Abstract In natural language settings, many interactions include more than two speakers, and real-life interpretation is based on all types of information available in all modalities. This constitutes a challenge for corpus-based analyses because the information in the audio and visual channels must be included in the coding. The goal of the DINLANG project is to tackle that challenge and analyze spontaneous interactions in family dinner settings (two adults and two to three children). The families use either French, or LSF (French sign language). Our aim is to compare how participants share language across the range of modalities found in vocal and visual languaging in coordination with dining. In order to pinpoint similarities and differences, we had to find a common coding tool for all situations (variations from one family to another) and modalities. Our coding procedure incorporates the use of the ELAN software. We created a template organized around participants, situations, and modalities, rather than around language forms. Spoken language transcription can be integrated, when it exists, but it is not mandatory. Data that has been created with another software can be injected in ELAN files if it is linked using time stamps. Analyses performed with the coded files rely on ELAN’s structured search functionalities, which allow to achieve fine-grained temporal analyses and which can be completed by using spreadsheets or R language. Keywords: multimodality, interaction, spoken language, sign language, ELAN, template, coding 1. Analyzing real-life interactions Language has long been studied out of its ecological context, first through written forms characterized by their linearity, then through invented sentences, and finally with a focus on speech, in experimental studies or semi- guided interactions. Even when gestures are integrated in the analyses, environments are most often stripped of objects or other activities whose affordances have a multitude of impacts on their use (but see Mondada, 2016). Those limits can be viewed as strengths as they have conducted to fruitful research on the language system. However, in order to capture the full complexity of language use, new approaches and methods are needed in which all our semiotic resources can be analyzed as they are deployed in their natural habitat involving the orchestration of bodies engaged in a variety of situated activities with a diversity of artifacts. The aim of our research is to capture language in its ecological environment in order to articulate its actional roots and its symbolic functions. Following Boutet (2018), we thus analyze the bodies of our participants as both the support (the instrument) and the …

Content truncated to 500 words