SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 774–790 Barcelona, Spain (Online), December 12, 2020. 774 SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets Parth Patwa1∗ Gustavo Aguilar2∗ Sudipta Kar2 Suraj Pandey3 Srinivas PYKL1 Bj¨orn Gamb¨ack4 Tanmoy Chakraborty3 Thamar Solorio2 Amitava Das5 1IIIT Sri City, India 2University of Houston, TX 3IIIT Delhi, India 4NTNU, Norway 5Wipro AI Labs, India 1{parthprasad.p17, srinivas.p}@iiits.in 2{gaguilaralas, skar3, tsolorio}@uh.edu 3{suraj18025, tanmoy}@iiitd.ac.in 4gamback@ntnu.no 5amitava.das2@wipro.com Abstract In this paper, we present the results of the SemEval-2020 Task 9 on Sentiment Analysis of Code- Mixed Tweets (SentiMix 2020).1 We also release and describe our Hinglish (Hindi-English) and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels. These corpora are comprised of 20K and 19K examples, re- spectively. The sentiment labels are - Positive, Negative, and Neutral. SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submit- ted systems to the Spanglish competition. The best performance achieved was 75.0% F1 score for Hinglish and 80.6% F1 for Spanglish. We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants. 1 Introduction The evolution of social media texts such as blogs, micro-blogs (e.g., Twitter), and chats (e.g., WhatsApp and Facebook messages) has created many new opportunities for information access and language tech- nologies. However, it has also posed many new challenges making it one of the current prime research areas in Natural Language Processing (NLP). Current language technologies primarily focus on English (Young, 2020), yet social media platforms demand methods that can also process other languages as they are inherently multilingual environments.2 Besides, multilingual communities around the world regularly express their thoughts in social media employing and alternating different languages in the same utterance. This mixing of languages, also known as code-mixing or code-switching,3 is a norm in multilingual societies and is one of the many NLP challenges that social media has facilitated. 1.1 Code-Mixing Challenges In addition to the writing aspects in social media, such as flexible grammar, permissive spelling, arbitrary punctuation, slang, and informal abbreviations (Baldwin et al., 2015; Eisenstein, 2013), code-mixing has introduced a diverse set of linguistic challenges. For instance, multilingual speakers tend to code-mix using a single alphabet regardless of whether the languages involved belong to different writing systems ∗Equal contribution. 1https://ritual-uh.github.io/sentimix2020/ This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/. 2Statistics show that half of the messages on Twitter are in a language other than English (Schroeder, 2010). 3We use code-mixing and code-switching interchangeably. 775 (i.e., language scripts). This behavior is known as transliteration, and code-mixers rely on the phonetic patterns of their writing (i.e., the actual sound) to convey their thoughts in the foreign language (i.e., the language adapted to a new script) (Sitaram et al., 2019). Another common pattern in code-mixing is the alternation of languages at the word level. This behavior often happens by inflecting words from one language with the rules of …

Content truncated to 500 words