Contextual Dependencies in Time-Continuous Multidimensional Affect Recognition

Contextual Dependencies in Time-Continuous Multidimensional Affect Recognition Dmitrii Fedotov 1,2, Denis Ivanko 1,2, Maxim Sidorov 1, Wolfgang Minker 1 1 Ulm University, 2 ITMO University Ulm Germany, St. Petersburg Russia {dmitrii.fedotov, wolfgang.minker}@uni-ulm.de, denis.ivanko11@gmail.com, maxim.sidorov@alumni.uni-ulm.de Abstract Modern research on emotion recognition often deals with time-continuously labelled spontaneous interactions. Such data is much closer to real world problems in contrast to utterance-level categorical labelling in acted emotion corpora that have widely been used to date. While working with time-continuous labelling, one usually uses context-aware models, such as recurrent neural networks. The amount of context needed to show the best performance should be defined in this case. Despite of the research done in this field there is still no agreement on this issue. In this paper we model different amounts of contextual input data by varying two parameters: sparsing coefficient and time window size. A series of experiments conducted with different modalities and emotional labels on the RECOLA corpora has shown a strong pattern between the amount of context used in model and performance. The pattern remains the same for different pairs of modalities and label dimensions, but the intensity differs. Knowledge about an appropriate context can significantly reduce the complexity of the model and increase its flexibility. Keywords: time-continuous affect recognition, affective context analysis, multimodal emotion recognition. 1. Introduction Real life human-human interaction consists of two main aspects: information contained in speech and emotion expressed by humans. Speech recognition techniques allow computers to understand human speech but they lack an emotional component. Exactly the same words or phrases may be a statement, a question or a guess if said with different emotions. These types of phrases should be recognised and processed by dialogue system differently. It is necessary for computers to understand human emotions in order to succeed in interaction with them. Modern research focus on natural interaction between computer-based systems and humans. These systems try to understand non-standardized questions and provide answers, similar to the spontaneous interaction between two humans. One of the most important parts of human understanding is the ability to identify and react to emotions. Emotion recognition may significantly improve the quality of human-computer interaction, speech recognition systems and artificial intelligence in general. Previous research on emotion recognition mostly dealt with utterance-level categorical data labelling, i.e. each data sample had one label from the list, e.g. anger, happiness, neutral, etc. However, recent research has focused on the dimensional time-continuous data that provides more flexibility and precision of emotion definition. This type of data requires more complex models and the definition of additional parameters, such as the amount of context to be used. Despite of the research conducted in this area, it is still an open question, how much previous data do the system need to provide the best performance. Studies on the effect of this parameter will help to build an effective end-to-end real time emotion recognition system. As shown in this paper, there is a strong correlation between the amount of context and the performance of an …

Content truncated to 500 words