DENS: A Dataset for Multi-class Emotion Analysis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6293–6298, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6293 DENS: A Dataset for Multi-class Emotion Analysis Chen Liu and Muhammad Osama and Anderson de Andrade Wattpad Toronto, ON, Canada cecilia, muhammad.osama, anderson@wattpad.com Abstract We introduce a new dataset for multi-class emotion analysis from long-form narratives in English. The Dataset for Emotions of Nar- rative Sequences (DENS) was collected from both classic literature available on Project Gutenberg and modern online narratives avail- able on Wattpad, annotated using Amazon Mechanical Turk. A number of statistics and baseline benchmarks are provided for the dataset. Of the tested techniques, we find that the fine-tuning of a pre-trained BERT model achieves the best results, with an av- erage micro-F1 score of 60.4%. Our results show that the dataset provides a novel opportu- nity in emotion analysis that requires moving beyond existing sentence-level techniques. 1 Introduction Humans experience a variety of complex emotions in daily life. These emotions are heavily reflected in our language, in both spoken and written forms. Many recent advances in natural language pro- cessing on emotions have focused on product re- views (McAuley et al., 2015) and tweets (Mo- hammad et al., 2018; Kant et al., 2018). These datasets are often limited in length (e.g. by the number of words in tweets), purpose (e.g. prod- uct reviews), or emotional spectrum (e.g. binary classification). Character dialogues and narratives in story- telling usually carry strong emotions. A memo- rable story is often one in which the emotional journey of the characters resonates with the reader. Indeed, emotion is one of the most important as- pects of narratives. In order to characterize narra- tive emotions properly, we must move beyond bi- nary constraints (e.g. good or bad, happy or sad). In this paper, we introduce the Dataset for Emo- tions of Narrative Sequences (DENS) for emotion analysis, consisting of passages from long-form fictional narratives from both classic literature and modern stories in English. The data samples con- sist of self-contained passages that span several sentences and a variety of subjects. Each sample is annotated by using one of 9 classes and an indi- cator for annotator agreement. 2 Background Using the categorical basic emotion model (Plutchik, 1979), (Mohammad and Kiritchenko, 2015; Mohammad, 2012) studied creating lexicons from tweets for use in emotion analysis. Recently, (Mohammad et al., 2018), (Klinger et al., 2018) and (Kant et al., 2018) proposed shared-tasks for multi-class emotion analysis based on tweets. Fewer works have been reported on understand- ing emotions in narratives. Emotional Arc (Rea- gan et al., 2016) is one recent advance in this direction. The work used lexicons and unsuper- vised learning methods based on unlabelled pas- sages from titles in Project Gutenberg1. For labelled datasets on narratives, (Alm et al., 2005) provided a sentence-level annotated cor- pus of childrens’ stories and (Kim and Klinger, 2018) provided phrase-level annotations on se- …

Content truncated to 500 words