GoEmotions: A Dataset of Fine-Grained Emotions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4040 GoEmotions: A Dataset of Fine-Grained Emotions Dorottya Demszky1∗ Dana Movshovitz-Attias2 Jeongwoo Ko2 Alan Cowen2 Gaurav Nemade2 Sujith Ravi3* 1Stanford Linguistics 2Google Research ddemszky@stanford.edu3Amazon Alexa {danama, jko, acowen, gnemade}@google.com sravi@sravi.org Abstract Understanding emotion expressed in language has a wide range of applications, from build- ing empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multi- ple downstream tasks. We introduce GoEmo- tions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demon- strate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with ex- isting emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT- based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.1 1 Introduction Emotion expression and detection are central to the human experience and social interaction. With as many as a handful of words we are able to express a wide variety of subtle and complex emotions, and it has thus been a long-term goal to enable machines to understand affect and emotion (Picard, 1997). In the past decade, NLP researchers made avail- able several datasets for language-based emotion classification for a variety of domains and appli- cations, including for news headlines (Strapparava and Mihalcea, 2007), tweets (CrowdFlower, 2016; Mohammad et al., 2018), and narrative sequences (Liu et al., 2019), to name just a few. However, ex- isting available datasets are (1) mostly small, con- taining up to several thousand instances, and (2) cover a limited emotion taxonomy, with coarse clas- ∗ Work done while at Google Research. 1Data and code available at https://github.com/ google-research/google-research/tree/ master/goemotions. Sample Text Label(s) OMG, yep!!! That is the final ans- wer. Thank you so much! gratitude, approval I’m not even sure what it is, why do people hate it confusion Guilty of doing this tbph remorse This caught me off guard for real. I’m actually off my bed laughing surprise, amusement I tried to send this to a friend but [NAME] knocked it away. disappointment Table 1: Example annotations from our dataset. sification into Ekman (Ekman, 1992b) or Plutchik (Plutchik, 1980) emotions. Recently, Bostan and Klinger (2018) have ag- gregated 14 popular emotion classification corpora under a unified framework that allows direct com- parison of the existing resources. Importantly, their analysis suggests annotation quality gaps in the largest manually annotated emotion classifi- cation dataset, CrowdFlower (2016), containing 40K tweets labeled for one of 13 emotions. While their work enables such comparative evaluations, it highlights the need for a large-scale, consistently labeled emotion dataset over a fine-grained taxon- omy, with demonstrated high-quality annotations. To this end, we compiled GoEmotions, the largest human annotated dataset of 58k carefully selected Reddit comments, labeled for …

Content truncated to 500 words