A «Portrait» Approach to Multichannel Discourse

A «Portrait» Approach to Multichannel Discourse Andrej A. Kibrik, Olga V. Fedorova Institute of Linguistics RAS & Lomonosov Moscow State University B. Kislovskij per., 1 & Leninskie Gory, 1, Moscow, Russia aakibrik@gmail.com, olga.fedorova@msu.ru Abstract This paper contributes to the research field of multichannel discourse analysis. Multimodal discourse analysis explores numerous channels involved in natural communication, such as verbal structure, prosody, gesticulation, facial expression, eye gaze, etc., and treats them as parts of an integral process. Among the key issues in multichannel studies is the question of the individual variation in multichannel behavior. We address this issue with the help of a multichannel resource “Russian Pear Chats and Stories” that is currently under construction (multidiscourse.ru). This corpus is based on a novel methodology of data collection and is produced with the help of state of the art technology including eyetracking. To address the issue of individual variation, we introduce the notion of a speaker’s individual portrait. In particular, we consider the Prosodic Portrait, the Oculomotor Portrait, and the Gesticulation Portrait. The proposed methodology is crucially important for fine-grained annotation procedures as well as for accurate statistic analyses of multichannel data. Keywords: multichannel discourse, prosody, gesticulation, eye gaze, a speaker's portrait 1.! Introduction People communicate with each other, using words, intonation, gestures, gaze, facial expression (Kress, 2002; Loehr, 2012; McNeill, 2005; Goldin-Meadow, 2014; Church et al. eds., 2017, inter alia), see Fig. 1. Figure 1: Spoken multichannel discourse. All of these communication channels are employed simultaneously and in conjunction with each other. Therefore, everyday human communication is a multichannel process. We are immersed in this communication throughout our lives, but still it remains an underexplored phenomenon. There are at least two causes leading to this situation. First, the process of multichannel communication is ephemeral and passes by without leaving material traces. Second, the content of various communication channels traditionally belongs to the competence of different disciplines, weakly connected with each other. In particular, the verbal component is studied by linguists, while gestures and eye movements are explored, primarily, by psychologists. Our research group set as its goal a comprehensive study of multichannel discourse, based on an integrated theoretical and methodological approach. A “multi-modal corpus” is defined as “an annotated collection of coordinated content on communication channels including speech, gaze, hand gesture and body language, and is generally based on recorded human behavior” (Foster, Oberlander, 2007: 307–308). As compared to monomodal corpora that already have a substantial history and tradition, multimodal corpora are still at their incipient stage. The most natural data have been assembled in the Fruit Carts Corpus that contains 240 videorecordings of 12 participants, each four to eight minutes long (Aist et al., 2012), the corpus D64, created for studies of everyday communication (Campbell, 2009), the InSight Interaction Corpus consisting of 15 recorded face-to-face conversations 20 min long each (Brône, Oben, 2015), as well as corpora created in the tradition of Conversation Analysis (Mondada, 2014; 2016, inter alia). When exploring any linguistic phenomenon, one addresses two opposite (and complementary) issues: …

Content truncated to 500 words