Unfolding the External Behavior and Inner Affective State of Teammates through Ensemble Learning: Experimental Evidence from a Dyadic Team Corpus

Unfolding the External Behavior and Inner Affective State of Teammates through Ensemble Learning: Experimental Evidence from a Dyadic Team Corpus Aggeliki Vlachostergiou1, Mark Dennison2, Catherine Neubauer2, Stefan Scherer3, Peter Khooshabeh2, Andre Harrison2 1Image,Video and Multimedia Systems Lab, National Technical University of Athens, Athens, Greece 2US Army Research Lab, Los Angeles, CA, USA 3Institute of Creative Technologies, University of Southern California, Los Angeles, CA, USA aggelikivl@image.ntua.gr, {mark.s.dennisson.ctr, catherine.e.neubauer2.ctr, peter.khooshabehadeh2.civ, andre.v.harrison2.civ}@mail.mil, scherer@ict.usc.edu Abstract The current study was motivated to understand the relationship between the external behavior and inner affective state of two team members (“instructor”-“defuser”) during a demanding operational task (i.e., bomb defusion). In this study we assessed team member’s verbal responses (i.e., length of duration) in relation to their external as well as internal affective cues. External behavioral cues include defuser’s verbal expressions while inner cues are based on physiological signals. More specifically, we differentiate between “defusers’” physiological patterns occurring after the “instructor’s” turns according to whether they belong to a short or a long turn-taking response interval. Based on the assumption that longer turn-taking behaviors are likely to be caused by demanding cognitive task events and/or stressful interactions, we hypothesize that inner mechanisms produced in these intense affective activity intervals will be reflected on defuser’s physiology. A dyadic team corpus was used to examine the association between the “defusers” physiological signals following the “instructor’s” questions to predict whether they occurred in a short or long turn-taking period of time. The results suggest that an association does exist between turn taking and inner affective state. Additionally, it was our goal to further unpack this association by creating diverse ensembles. As such, we studied various base learners and different ensemble sizes to determine the best approach towards building a stable diverse ensemble that generalizes well on the external and inner cues of individuals. Keywords: Turn-takings, Physiological features, MEAP, Dyadic Team Corpus, Ensemble Learning 1. Introduction Lack of emotional expressivity is one of the main deficits that characterizes periods of stress when team members perform highly cognitive cooperative tasks. In cases such as this, teammates may find it more difficult to express their conscious feelings and show different patterns in perceiv- ing and conveying emotional information when working together to meet a common goal(Jones and George, 1998; Prati et al., 2003). In light of these observations, having a way to monitor the internal state of teammates within such contexts might provide us new insights with respect to the mechanisms of their interaction and affectivity. During high workload and high stress tasks, the sympa- thetic nervous system is accountable for activating glands and organs that are responsible for defending the body from perceived threats. This activation is associated with changes in arousal that are further influenced by emotion, cognition or attention. Stress results in increased sympa- thetic activity and can be tracked for example through bod- ily reactions, such as an increase in heart rate, greater blood flow to extremities and an increase in the respiration rate etc. Thus, a combination of more than …

Content truncated to 500 words