A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set

A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set Ian D. Wood, John P. McCrae, Vladimir Andryushechkin, Paul Buitelaar Insight Centre for Data Analytics, National University of Ireland Galway name.surname@insight-centre.org Abstract While the recognition of positive/negative sentiment in text is an established task with many standard data sets and well developed methodologies, the recognition of more nuanced affect has received less attention, and in particular, there are very few publicly available gold standard annotated resources. To address this lack, we present a series of emotion annotation studies on tweets culminating in a publicly available collection of 2,019 tweets with scores on four emotion dimensions: valence, arousal, dominance and surprise, following the emotion representation model identified by Fontaine et.al. (Fontaine et al., 2007). Further, we make a comparison of relative vs. absolute annotation schemes. We find improved annotator agreement with a relative annotation scheme (comparisons) on a dimensional emotion model over a categorical annotation scheme on Ekman’s six basic emotions (Ekman et al., 1987), however when we compare inter-annotator agreement for comparisons with agreement for a rating scale annotation scheme (both with the same dimensional emotion model), we find improved inter-annotator agreement with rating scales, challenging a common belief that relative judgements are more reliable. Keywords: emotion, annotation, annotator-agreement, social-media, affective-computing 1. Introduction Beyond simple positive/negative sentiment, there are two main emotion representation schemes that have been used in automated emotion recognition research. The first posits the existence of “basic” emotions or emotion categories to which all other emotions belong, for example the six emo- tions categories identified by Ekman (Ekman et al., 1987): joy, fear, anger, disgust, sadness and surprise. The sec- ond envisages each emotional state as a point in a dimen- sional space, each dimension corresponding to a character- istic of the emotion, the most widely used model presenting three dimensions (Osgood et al., 1975; Russell and Mehra- bian, 1977): Valence (also termed evaluation-pleasantness), Dominance (also termed potency-control) and Arousal (also termed activation-arousal, a level of physical energy and action tendency). The question of which dimensions most effectively capture variation in emotional states was recently re-opened by Fontaine et.al. (2007), who identi- fied those same three dimensions (though with higher im- portance attributed to Dominance) with one extra: Surprise or unpredictability. Research effort in the recognition of affect in text has fo- cussed to a large extent on recognition of positive/negative sentiment, while more nuanced emotion representation models have received relatively little attention. In particu- lar, there has been a lack of quality annotated resources for model building and evaluation in that space (Mohammad, 2016) and in particular with dimensional annotations. Ex- isting text corpora with dimensional emotion annotations include Affective Norms for English Texts (Bradley and Lang, 2007), a collection of 120 generic texts with VAD an- notations; a collection of 2,895 Facebook posts annotated by just two annotators with valence and arousal dimen- sions (Preotiuc-Pietro et al., 2016). Yu et.al. (2016) pre- sented a collection of 2009 Chinese sentences from various online texts, …

Content truncated to 500 words