Convolutional and Recurrent Neural Networks for Spoken Emotion Recognition

Convolutional and Recurrent Neural Networks for Spoken Emotion Recognition Aaron Keesing School of Computer Science University of Auckland New Zealand akee511@aucklanduni.ac.nz Ian Watson School of Computer Science University of Auckland New Zealand ian@cs.auckland.ac.nz Michael Witbrock School of Computer Science University of Auckland New Zealand m.witbrock@auckland.ac.nz Abstract We test four models proposed in the speech emotion recognition (SER) literature on 15 public and academic licensed datasets in speaker-independent cross-validation. Results indicate differences in the performance of the models which is partly dependent on the dataset and features used. We also show that a standard utterance-level feature set still per- forms competitively with neural models on some datasets. This work serves as a starting point for future model comparisons, in addi- tion to open-sourcing the testing code. 1 Introduction Speech emotion recognition (SER) is the analy- sis of speech to predict the emotional state of the speaker, for which there are many current and potential applications (Peter and Beale, 2008; Koolagudi and Rao, 2012). As speech-enabled devices become more prevalent, the need for reli- able and robust SER increases, and also the need for comparability of results on common datasets. While there has been a large amount of research in this field, a lot of results come from testing only on one or two datasets, which may or may not be pub- licly available. Additionally, different methodolo- gies are often used, reducing direct comparability of results. Given the wide variety of neural architec- tures and testing methodologies, there is need for a common testing framework to help comparisons. This study aims to test some SER models pro- posed in the literature on a discrete emotion classi- fication task, and promote reproducibility of results by using public and academic licensed datasets. In addition, the code is publicly hosted on GitHub1 under an open source license, so that our results may be verified and built upon. Our work has two 1https://github.com/Broad-AI-Lab/ emotion main benefits. First, it serves as a baseline refer- ence for future research that uses datasets present in this study. Second, it allows for comparisons be- tween datasets to see which of their properties may influence classification performance of different models. The paper is structured as follows. In Section 2 related work is given, and in Section 3 we list the datasets used in this study. The tested methods are outlined in Section 4, and the results given in Section 5. We briefly discuss these results in Section 6 and a conclude in Section 7. 2 Related Work There has been some previous work in compar- ing SER techniques on a number of datasets. In Schuller et al. (2009a), Schuller et al. compare a hidden Markov model/Gaussian mixture model (HMM/GMM) and a SVM classifier for emo- tion class, arousal and valence prediction on nine datasets. For HMM/GMM, 12 MFCC, log-frame- energy, speed and acceleration features, are ex- tracted per frame. For SVM, 6552 features are extracted based on 39 statistical functionals of 56 low-level descriptors (LLDs). Testing was done in a leave-one-speaker-out …

Content truncated to 500 words