From analysis to modeling of engagement as sequences of multimodal behaviors

From Analysis to Modeling of Engagement as Sequences of Multimodal Behaviors Soumia Dermouche, Catherine Pelachaud CNRS UMR 7222, ISIR, Sorbonne Universit´es 4 Place Jussieu Paris, France {dermouche, pelachaud}@isir.upmc.fr Abstract In this paper, we present an approach to endow an Embodied Conversational Agent with engagement capabilities. We relied on a corpus of expert-novice interactions. Two types of manual annotation were conducted: non-verbal signals such as gestures, head movements and smiles; engagement level of both expert and novice during the interaction. Then, we used a temporal sequence mining algorithm to extract non-verbal sequences eliciting variation of engagement perception. Our aim is to apply these findings in human-agent interaction to analyze user’s engagement level and to control agent’s behavior. The novelty of this study is to consider explicitly engagement as sequence of multimodal behaviors. Keywords: Engagement, Non-verbal behavior, ECA, Temporal Sequence Mining, Human-agent interaction 1. Introduction Embodied Conversational Agents (ECA) are virtual charac- ters that can interact with a user. Today, ECAs are increas- ingly being integrated in our everyday life, for example, for training, social coaching, and science teaching (Graesser et al., 2007). Our work is part of the H2020 European project ARIA-VALUSPA (Valstar et al., 2016) that aims to build an ECA able to play the role of an expert and to share its domain knowledge with a novice user. In this project, we focus on an important aspect of human-agent interaction, namely, engagement that ensures the interaction to go on. A survey of engagement definition in human-agent interaction is given in (Glas and Pelachaud, 2015). Engagement can be defined as: “the value that a participant in an interaction attributes to the goal of being together with the other par- ticipant(s) and of continuing the interaction”(Poggi, 2007). Engagement is also defined as “the process by which par- ticipants involved in an interaction start, maintain and ter- minate an interaction” (Sidner et al., 2005). Engagement is not measured from single cues, but rather from several cues that arise over a certain time window (Peters et al., 2005). The goal of this work is twofold: (1) user’s engagement detection: the ECA should be able to detect, in real time, the engagement level of the user. (2) ECA’s engagement modeling: the ECA should adapt its behavior in order to maintain the desired level of engagement during the inter- action. Specifically, this work investigates what are the multimodal behaviors that participate to a change of per- ception of the engagement level in a human-agent interac- tion. For this, we rely on sequence mining algorithms to associate user’s and agent’s non-verbal behaviors with dif- ferent engagement levels. Several works focused on associating verbal and non-verbal behaviors with engagement in human-agent interaction, but most of them are limited to few signals such as smiling and head nod (Allwood and Cerrato, 2003; Castellano et al., 2009). In our work, we consider a set of non-verbal modal- ities (gesture, head movement and directions, smiling, etc.) jointly with their temporal synchronization (order, starting time and duration). Our study is performed …

Content truncated to 500 words