XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection

Proceedings of the 28th International Conference on Computational Linguistics, pages 6542–6552 Barcelona, Spain (Online), December 8-13, 2020 6542 XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection Emily ¨Ohman and Marc P`amies and Kaisla Kajava and J¨org Tiedemann University of Helsinki firstname.lastname@helsinki.fi Abstract We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of human- annotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 ad- ditional languages, providing new resources for many low-resource languages. We use Plutchik’s core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset. The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection. 1 Introduction There is an ever increasing need for labeled datasets for machine learning. This is true for English as well as other, often under-resourced, languages. We provide a cross-lingual fine-grained sentence-level emotion and sentiment dataset. The dataset consists of parallel manually annotated data for English and Finnish, with additional parallel datasets of varying sizes for a total of 32 languages created by annotation projection. We use Plutchik’s Wheel of Emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) (Plutchik, 1980) as our annotation scheme with the addition of neutral on movie subtitle data from OPUS (Lison and Tiedemann, 2016). We perform evaluations with fine-tuned cased multilingual and language specific BERT (Bidirectional Encoder Representations from Transformers) models (Devlin et al., 2019), as well as Suport Vector Machines (SVMs). Our evaluations show that the human-annotated datasets behave on par with compa- rable state-of-the-art datasets such as the GoEmotions dataset (Demszky et al., 2020). Furthermore, the projected datasets have accuracies that closely resemble human-annotated data with macro f1 scores of 0.51 for the human annotated Finnish data and 0.45 for the projected Finnish data when evaluating with FinBERT (Virtanen et al., 2019). The XED dataset can be used in emotion classification tasks and other applications that can benefit from sentiment analysis and emotion detection such as offensive language identification. The data is open source1 licensed under a Creative Commons Attribution 4.0 International License (CC-BY). In the following sections we discuss related work and describe our datasets. The datasets are then evaluated and the results discussed in the discussion section. 2 Background & Previous Work Datasets created for sentiment analysis have been available for researchers since at least the early 2000s (M¨antyl¨a et al., 2018). Such datasets generally use a binary or ternary annotation scheme (positive, negative + neutral) (e.g. Blitzer et al. (2007)) and have traditionally been based on review data such as, e.g. Amazon product reviews, or movie reviews (Blitzer et al., 2007; Maas et al., 2011; Turney, 2002). Many, if not most, emotion datasets on the other hand use Twitter as a source and individual tweets as 1https://github.com/Helsinki-NLP/XED This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/. 6543 level …

Content truncated to 500 words