ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8770 - 8785 December 7-11, 2022 ©2022 Association for Computational Linguistics ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture Youssef Mohamed1∗ Mohamed Abdelfattah 1 Shyma Alhuwaider1 Feifan Li 1 Xiangliang Zhang 2 Kenneth Ward Church 3 Mohamed Elhoseiny1* 1KAUST 2 University of Notre Dame 3 Northeastern University {youssef.mohamed,mohamed.abdelfattah,shyma.alhuwaider,feifan.li}@kaust.edu.sa xzhang33@nd.edu, k.church@northeastern.edu, mohamed.elhoseiny@kaust.edu.sa a) ﻮﻤﻨﻟا ﺮﻋﺎﺸﻣ .ﻞﯿﻤﺟ ﻲﻌﯿﺒط لﻼﺷ .ةدﻮﺟﻮﻣ ﺔﻗﺎﻄﻟاو ﺔﯾﻮﯿﺤﻟاو Translation: Beautiful natural waterfall. Feelings of growth, vitality and energy. The water that's rushing downward looks like a bride's wedding veil. 瀑布就像四蹄生风的白马如潮水 涌来，非常的壮观 Translation: The waterfall is like a white horse and wind, it is spectacular. b) Translation: Girls sitting with their mother outside the house, exchanging love and affection, pigeons flying over a tree. The women relaxing while birds are flying about makes me feel relaxed and calm as well. Translation: Three sisters lying on a bench and watching the birds fly comfortably. c) Translation: The use of black and white for painting the forests with all its details brings out a feeling of satisfaction. The trees are dead and exposing their roots due to erosion and lack of water. Translation: After the snow in winter, there is snow everywhere, and the dead trees look very depressed. Excitement Arabic Awe English Contentment Chinese Contentment Arabic Contentment Arabic Sadness English Contentment English Contentment Chinese Sadness Chinese Figure 1: ArtELingo, a multilingual dataset and benchmark of WikiArt with captions & emotions Abstract This paper introduces ArtELingo, a new bench- mark and dataset, designed to encourage work on diversity across languages and cultures. Fol- lowing ArtEmis, a collection of 80k artworks from WikiArt with 0.45M emotion labels and English-only captions, ArtELingo adds another 0.79M annotations in Arabic and Chinese, plus 4.8K in Spanish to evaluate “cultural-transfer” performance. More than 51K artworks have 5 annotations or more in 3 languages. This diver- sity makes it possible to study similarities and differences across languages and cultures. Fur- ther, we investigate captioning tasks, and find diversity improves the performance of baseline models. ArtELingo is publicly available1 with standard splits and baseline models. We hope our work will help ease future research on mul- tilinguality and culturally-aware AI. ∗ Corresponding Authors 1www.artelingo.org 1 Introduction Figure 1 compares and contrasts annotations on WikiArt across language/culture. We believe these differences are interesting and important, and far from random. One might suggest using machine translation to translate English captions to many other languages, but we believe that doing so would miss much of the opportunity. Building human- compatible AI that is more aware of our emotional being is important for increasing the social accep- tance of AI. ArtEmis (Achlioptas et al., 2021) is an important step in this direction, introducing a collection of 0.45M emotion labels and affective language explanations in English on more than 80,000 artworks from WikiArt. However, by de- sign, ArtEmis is limited to English, lacking cover- age of other cultures and languages. Cultural differences are …

Content truncated to 500 words