A Multimodal Motion-Captured Corpus of Matched and Mismatched Extravert-Introvert Conversational Pairs

A Multimodal Motion-Captured Corpus of Matched and Mismatched Extravert-Introvert Conversational Pairs Jackson Tolins1, Kris Liu1, Yingying Wang2, Jean E. Fox Tree1, Marilyn Walker3, Michael Neff2 1University of California, Santa Cruz, Department of Psychology 2University of California, Davis, Department of Computer Science 3University of California, Santa Cruz, Department of Computer Science E-mail: jtolins@ucsc.edu, kyliu@ucsc.edu, yiwang@ucdavis.edu, foxtree@ucsc.edu, mawalker@ucsc.edu, mpneff@ucdavis.edu Abstract This paper presents a new corpus, the Personality Dyads Corpus, consisting of multimodal data for three conversations between three personality-matched, two-person dyads (a total of 9 separate dialogues). Participants were selected from a larger sample to be 0.8 of a standard deviation above or below the mean on the Big-Five Personality extraversion scale, to produce an Extravert-Extravert dyad, an Introvert-Introvert dyad, and an Extravert-Introvert dyad. Each pair carried out conversations for three different tasks. The conversations were recorded using optical motion capture for the body and data gloves for the hands. Dyads’ speech was transcribed and the gestural and postural behavior was annotated with ANVIL. The released corpus includes personality profiles, ANVIL files containing speech transcriptions and the gestural annotations, and BVH files containing body and hand motion in 3D. The corpus should be a useful resource for researchers working on generating human-like and adaptive multimodal behaviors in intelligent virtual agents. Keywords: spontaneous communication, gesture production, personality, motion capture, data gloves, entrainment, nonverbal behavior generation 1. Background and Motivation This paper presents a new corpus, the Personality Dyads Corpus, consisting of multimodal data for three conversations each between three two-person dyads (a total of nine dialogues across three sessions), two matched for personality and one mismatched. The goal of the corpus is to support research on how personality and entrainment affect conversational behavior. It is well known that the behavior of people engaged in dialogue is influenced both by their individual personalities and by their joint participation in the collaborative act of communication. However strongly individuality may be conveyed by movement, gesture, and linguistic style, conversational participants dynamically adapt to their conversational partner. Theories that attempt to explain such adaptation include the collaborative theory of language use (Clark, 1996; Schober, 1998), theories of alignment (Branigan, Pickering, Pearson, & McLean, 2010), communication accommodation theory (Tickle-Degnen & Rosenthal, 1990; Giles, Taylor, & Bourhis, 1973; Giles, Coupland, & Coupland, 1991), and narrative theories of identity in interaction (Thorne, 1987; McAdams & Pals, 2006). Entrainment arises out of interlocutors grounding communication with each other. As a conversation develops, conversational participants collaborate on conversational goals, the meanings they are communicating with each other, what perspective to take in approach to a situation, how to refer to topics in joint attention, and other things. For example, participants may create conceptual pacts on how to refer to an object, agreeing to talk about the same object as either a shoe, a loafer, or a brown loafer (Brennan & Clark, 1996) or show preference for gestures more like their own (Luo, Ng-Thow-Hing and Neff, 2013). Language and gesture are thought to be closely tied to one another, linked both temporally …

Content truncated to 500 words