Modeling Collaborative Multimodal Behavior in Group Dialogues: The MULTISIMO Corpus

Modeling Collaborative Multimodal Behavior in Group Dialogues: The MULTISIMO Corpus Maria Koutsombogera, Carl Vogel Trinity College Dublin College Green, Dublin 2, Ireland {koutsomm, vogel}@cs.tcd.ie Abstract We present a multimodal corpus that has been recently developed within the MULTISIMO project and targets the investigation and modeling of collaborative aspects of multimodal behavior in groups that perform simple tasks. The corpus consists of a set of human-human interactions recorded in multiple modalities. In each interactive session two participants collaborate with each other to solve a quiz while assisted by a facilitator. The corpus has been transcribed and annotated with information related to verbal and non-verbal signals. A set of additional annotation and processing tasks are currently in progress. The corpus includes survey materials, i.e. personality tests and experience assessment questionnaires filled in by all participants. This dataset addresses multiparty collaborative interactions and aims at providing tools for measuring collaboration and task success based on the integration of the related multimodal information and the personality traits of the participants, but also at modeling the multimodal strategies that members of a group employ to discuss and collaborate with each other. The corpus is designed for public release. Keywords: multimodal corpus, multiparty interaction, collaboration, task success, social behavior, personality 1. Introduction The development of multimodal corpora is an essential step in the investigation of the human behavior, as the knowl- edge reflected in them can lead to models of social be- havior to be integrated in intelligent interfaces, making the human-computer interaction as effective, natural and in- tuitive as possible. Multimodal corpora provide informa- tion about the way different expressive modalities shape the structure of the interaction (i.e. turn management) and convey the speakers’ cognitive and affective state in any given moment (including feedback responses and emo- tions), thus demonstrating the speakers’ interactional and social behavior (Vinciarelli et al., 2015; Esposito et al., 2015). While two-party interaction is already a rich and informative setup, multiparty interaction is even more chal- lenging because of the dynamics developed among group members (Gatica-Perez et al., 2017). In recent years, various multimodal and multiparty corpora have been created to study and analyse human behavior and collaboration aspects of it, in two-party (HCRC Map Task Corpus (Thompson et al., 1993)) and in group interaction settings, such as the AMI corpus (Carletta et al., 2005), the Mission Survival Corpus-2 (Mana et al., 2007), the Canal 9 political debates satabase (Vinciarelli et al., 2009) and the Idiap Wolf Corpus (Hung and Chittaranjan, 2010), to name a few. The importance of analyzing such corpora lies in decoding communicative patterns involving verbal and non-verbal modalities. By definition, group dialogue is a canvas where different communicative intentions, per- sonalities, lexical choices that may affect the outcome and the effectiveness of the interaction are manifested by the participants. In terms of behavior modelling, efforts focus on automatically analyzing various facets of group inter- actions and collecting this knowledge to improve the qual- ity of the interaction either in human-human or in human- machine settings. Related work …

Content truncated to 500 words