SemEval-2018 Task 1: Affect in Tweets

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1–17 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 1: Affect in Tweets Saif M. Mohammad National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Felipe Bravo-Marquez The University of Waikato, New Zealand fbravoma@waikato.ac.nz Mohammad Salameh Carnegie Mellon University in Qatar msalameh@qatar.cmu.edu Svetlana Kiritchenko National Research Council Canada svetlana.kiritchenko@nrc-cnrc.gc.ca Abstract We present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks on inferring the affectual state of a person from their tweet. For each task, we created labeled data from English, Arabic, and Spanish tweets. The individual tasks are: 1. emotion intensity regression, 2. emotion intensity ordinal classi- fication, 3. valence (sentiment) regression, 4. valence ordinal classification, and 5. emotion classification. Seventy-five teams (about 200 team members) participated in the shared task. We summarize the methods, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful. We also analyze systems for consistent bias towards a particular race or gender. The data is made freely available to further improve our understanding of how peo- ple convey emotions through language. 1 Introduction Emotions are central to language and thought. They are familiar and commonplace, yet they are complex and nuanced. Humans are known to per- ceive hundreds of different emotions. Accord- ing to the basic emotion model (aka the categor- ical model) (Ekman, 1992; Plutchik, 1980; Par- rot, 2001; Frijda, 1988), some emotions, such as joy, sadness, and fear, are more basic than others—physiologically, cognitively, and in terms of the mechanisms to express these emotions. Each of these emotions can be felt or expressed in varying intensities. For example, our ut- terances can convey that we are very angry, slightly sad, absolutely elated, etc. Here, inten- sity refers to the degree or amount of an emo- tion such as anger or sadness.1 As per the valence–arousal–dominance (VAD) model (Rus- sell, 1980, 2003), emotions are points in a 1Intensity is different from arousal, which refers to the extent to which an emotion is calming or exciting. three-dimensional space of valence (positiveness– negativeness), arousal (active–passive), and domi- nance (dominant–submissive). We use the term af- fect to refer to various emotion-related categories such as joy, fear, valence, and arousal. Natural language applications in commerce, public health, disaster management, and public policy can benefit from knowing the affectual states of people—both the categories and the intensities of the emotions they feel. We thus present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks where automatic systems have to infer the affectual state of a person from their tweet.2 We will refer to the author of a tweet as the tweeter. Some of the tasks are on the intensities of four basic emotions common to many proposals of basic emotions: anger, fear, joy, and sadness. Some of the tasks are on valence or sentiment intensity. Finally, we include an emotion classification task over eleven emotions commonly expressed …

Content truncated to 500 words