MUSER: MUltimodal Stress detection using Emotion Recognition as an Auxiliary Task

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2714–2725 June 6–11, 2021. ©2021 Association for Computational Linguistics 2714 MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task Yiqun Yao1, Michalis Papakostas1, Mihai Burzo2, Mohamed Abouelenien3, Rada Mihalcea1 1Computer Science and Engineering, University of Michigan 2Mechanical Engineering, University of Michigan 3Computer and Information Science, University of Michigan {yaoyq,mpapakos,mburzo,zmohamed,mihalcea}@umich.edu Abstract The capability to automatically detect human stress can benefit artificial intelligent agents involved in affective computing and human- computer interaction. Stress and emotion are both human affective states, and stress has proven to have important implications on the regulation and expression of emotion. Al- though a series of methods have been estab- lished for multimodal stress detection, limited steps have been taken to explore the underly- ing inter-dependence between stress and emo- tion. In this work, we investigate the value of emotion recognition as an auxiliary task to im- prove stress detection. We propose MUSER – a transformer-based model architecture and a novel multi-task learning algorithm with speed-based dynamic sampling strategy. Eval- uations on the Multimodal Stressed Emotion (MuSE) dataset show that our model is effec- tive for stress detection with both internal and external auxiliary tasks, and achieves state-of- the-art results. 1 Introduction Stress is a feeling of emotional or physical tension, as a response to the environment when people have difficulty dealing with the conditions (Dobson and Smith, 2000; Muthukumar and Nachiappan, 2013). Stress detection is a classification task that predicts whether a certain target is under stress. The task has drawn research attention for two reasons: first, stress detection plays an important role in applica- tions related to psychological well-being (Cohen et al., 1991), cognitive behavior therapies (Tull et al., 2007), and safe driving (Gao et al., 2014; Chen et al., 2017); second, stress is a known reg- ulator of human emotion mechanisms (Tull et al., 2007), and thus research on stress detection can potentially benefit the development of emotionally intelligent agents. The impact of stress on human behavior can be observed through various modalities. Previ- ous work has considered both unimodal and mul- timodal stress detection using acoustic, video and physiological sensor signals (Lane et al., 2015; Jaques et al., 2016; Aigrain et al., 2016; Alberdi et al., 2016; Bara et al., 2020). However, text-based stress detection remains vastly underexplored, with some studies (Lin et al., 2014) showing the poten- tial for further research. In recent years, the surge of advanced natural language understanding mod- els and structures provides a great opportunity for stress detection systems, especially using the tex- tual modality. In this work, we focus on the textual and acoustic modalities. For the model architec- ture, we use Transformers (Vaswani et al., 2017) as a textual encoder and Multi-Layer Perceptrons (MLP) as an acoustic encoder. The majority of existing stress detection meth- ods are based on single-task learning with the bi- nary stress/non-stress labels. However, stress is not an isolated affective state, …

Content truncated to 500 words