Modeling Emotion Dynamics in Song Lyrics with State Space Models

Modeling Emotion Dynamics in Song Lyrics with State Space Models Yingjin Song∗ Department of Information and Computing Sciences Utrecht University, Netherlands y.song5@uu.nl Daniel Beck School of Computing and Information Systems University of Melbourne, Australia d.beck@unimelb.edu.au Abstract Most previous work in music emotion recog- nition assumes a single or a few song-level labels for the whole song. While it is known that different emotions can vary in intensity within a song, annotated data for this setup is scarce and difficult to obtain. In this work, we propose a method to predict emotion dynam- ics in song lyrics without song-level super- vision. We frame each song as a time series and employ a State Space Model (SSM), com- bining a sentence-level emotion predictor with an Expectation-Maximization (EM) procedure to generate the full emotion dynamics. Our experiments show that applying our method consistently improves the performance of sentence-level baselines without requiring any annotated songs, making it ideal for lim- ited training data scenarios. Further analysis through case studies shows the benefits of our method while also indicating the limitations and pointing to future directions. 1 Introduction Music and emotions are intimately connected, with almost all music pieces being created to ex- press and induce emotions (Juslin and Laukka, 2004). As a key factor of how music conveys emotion, lyrics contain part of the semantic infor- mation that the melodies cannot express (Besson et al., 1998). Lyrics-based music emotion recog- nition has attracted increasing attention driven by the demand to process massive collections of songs automatically, which is an important task for streaming and media service providers (Kim et al., 2010; Malheiro et al., 2016; Agrawal et al., 2021). Vanilla emotion recognition studies in Natu- ral Language Processing (NLP) assume the text instance expresses a static and single emotion ∗Work done when the first author was at The University of Melbourne. (Mohammad and Bravo-M´arquez, 2017; Nozza et al., 2017; Mohammad et al., 2018). How- ever, emotion is non-static and highly correlated with the contextual information, making the single-label assumption too simplistic in dynamic scenarios, not just in music (Schmidt and Kim, 2011) but also in other domains such as conver- sations (Poria et al., 2019b). Figure 1 shows an example of this dynamic behavior, where the in- tensities of three different emotions vary within a song. Accurate emotion recognition systems should ideally generate the full emotional dynam- ics for each song, as opposed to simply predicting a single label. A range of datasets and corpora for modeling dynamic emotion transitions has been developed in the literature (McKeown et al., 2011; Li et al., 2017; Hsu et al., 2018; Poria et al., 2019a; Firdaus et al., 2020), but most of them do not use song lyrics as the domain and assume discrete, cat- egorical labels for emotions (either the pres- ence or absence of one emotion). To the best of our knowledge, the dataset from Mihalcea and Strapparava (2012) is the only one that provides full fine-grained emotion intensity annotations for song lyrics at the …

Content truncated to 500 words