A Verbal and Gestural Corpus of Story Retellings to an Expressive Embodied Virtual Character

A Verbal and Gestural Corpus of Story Retellings to an Expressive Embodied Virtual Character Jackson Tolins1, Kris Liu1, Michael Neff3, Marilyn Walker2, Jean E. Fox Tree1 1University of California, Santa Cruz, Department of Psychology 2University of California, Santa Cruz, Department of Computer Science 3University of California, Davis, Department of Computer Science E-mail: jtolins@ucsc.edu, kyliu@ucsc.edu, foxtree@ucsc.edu, ma, M@ucsc.edu, mpneff@ucdavis.edu Abstract We present a corpus of 44 human-agent verbal and gestural story retellings designed to explore whether humans would gesturally entrain to an embodied intelligent virtual agent. We used a novel data collection method where an agent presented story components in installments, which the human would then retell to the agent. At the end of the installments, the human would then retell the embodied animated agent the story as a whole. This method was designed to allow us to observe whether changes in the agent’s gestural behavior would result in human gestural changes. The agent modified its gestures over the course of the story, by starting out the first installment with gestural behaviors designed to manifest extraversion, and slowly modifying gestures to express introversion over time, or the reverse. The corpus contains the verbal and gestural transcripts of the human story retellings. The gestures were coded for type, handedness, temporal structure, spatial extent, and the degree to which the participants’ gestures match those produced by the agent. The corpus illustrates the variation in expressive behaviors produced by users interacting with embodied virtual characters, and the degree to which their gestures were influenced by the agent’s dynamic changes in personality-based expressive style. Keywords: spontaneous communication, gesture production, virtual agents, personality, stylistic adaptation, gestural and linguistic paraphrase 1. Background and Motivation We present a corpus of 44 human-virtual agent story retellings that were designed to collect both verbal and nonverbal aspects of human-agent interaction in order to investigate whether people would gesturally entrain to a virtual agent. We used a novel method involving retelling a story in installments in order to investigate the effect on human gestural and verbal behavior of a protocol in which the intelligent embodied agent changed its gestural expressive style over time, either starting as introverted and becoming extraverted, or vice versa. Previous research suggests that intelligent virtual agents will be viewed more positively if they are both capable of nonverbal expressive behavior, and capable of adapting their behavior to their conversational partner. This adaptation may be based on personality, specific behaviors of the partner, emotional expressiveness or matching, or cultural norms (Andre et al 2014, Endrass et al, 2010; Kopp et al 2006, Mairesse & Walker 2010, 2011; Salem et al, 2012, Hartmann et al 2005). Some studies have suggested that agents that can behaviorally coordinate will be viewed more positively, are more believable, and are more persuasive (Bailenson, et al., 2008, Tapus & Mataric, 2008, Andre et al 2000). People respond to coordinating computers in ways that are similar to how they respond to coordinating people. If the computer mimics their own verbal personality style, people like the …

Content truncated to 500 words