Exploring Reliability of Gold Labels for Emotion Detection in Twitter

Proceedings of Recent Advances in Natural Language Processing, pages 1350–1359 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_151 1350 Exploring Reliability of Gold Labels for Emotion Detection in Twitter Sanja ˇStajner Symanto Research Nuremberg, Germany sanja.stajner@symanto.com Abstract Emotion detection from social media posts has attracted noticeable attention from natural lan- guage processing (NLP) community in recent years. The ways for obtaining gold labels for training and testing of the systems for au- tomatic emotion detection differ significantly from one study to another, and pose the ques- tion of reliability of gold labels and obtained classification results. This study systemati- cally explores several ways for obtaining gold labels for Ekman’s emotion model on Twitter data and the influence of the chosen strategy on the manual classification results. 1 Introduction Interest for automatic emotion detection has been gaining popularity in the last ten years (Acheam- pong et al., 2020, Figures 3 and 4). The span of its applications ranges from empathetic chatbots and virtual agents (Paiva et al., 2017; Rashkin et al., 2019; Lin et al., 2019b; Shin et al., 2019; Lin et al., 2019a; Ma et al., 2020) to social media and public opinion analysis (e.g. (Anstead and O’Loughlin, 2014; Wu et al., 2020; Loureiro and All´o, 2020)). Nevertheless, the task proved challenging, espe- cially when attempted at purely textual utterances as opposed to the multi-modal ones (Poria et al., 2019), probably due to missing visual and audio cues (Acheampong et al., 2020). Previous studies reported some of the challenges in automatic emotion detection from texts: differ- ent perspectives one may take (Buechel and Hahn, 2017b; Alm et al., 2005), missing context ( ¨Ohman et al., 2020; Mohammad, 2012), non-literal mean- ing (Mohammad, 2012), high subjectivity of the task and low inter-annotator agreement even among trained annotators (Alm et al., 2005; Schuff et al., 2017). For example, the utterance “Italy defeats France in World Cup Final” (Katz et al., 2007) is most probably neutral from the writer’s (journal- ist’s) perspective, while evoking strong and prob- ably opposite emotions among Italian and French readers (Buechel and Hahn, 2017b). The utter- ance “Time for shopping” might be neutral, or ex- press/evoke various emotions (e.g. joy, anger, fear) depending on the writer’s/reader’s associations and personal experiences with shopping. The field of emotion detection from text, similar to many other areas of natural language process- ing, suffers from the absence of standards for hu- man annotation, and systematic investigations of how different strategies for obtaining gold labels influence classification results. Notable exceptions are the studies by Mohammad and Turney (2013) and Buechel and Hahn (2017b). Mohammad and Turney (2013) found that asking annotators which emotion is the word associated with yields higher inter-annotator agreement than asking them which emotion the word evokes. This result indicated that annotating emotions from text’s perspective is less subjective than annotating them from reader’s per- spective. Motivated by those results, Buechel and Hahn (2017b) investigated the influence of perspec- tive on the inter-annotator agreement in emotion annotation at a sentence level. A recent study …

Content truncated to 500 words