Twitter corpus of Resource-Scarce Languages for Sentiment Analysis and Multilingual Emoji Prediction

Twitter corpus of Resource-Scarce Languages for Sentiment Analysis and Multilingual Emoji Prediction Nurendra Choudhary, Rajat Singh, Vijjini Anvesh Rao, Manish Shrivastava Language Technologies Research Center, Kohli Center on Intelligent Systems, International Institute of Information Technology, Hyderabad, India {nurendra.choudhary, rajat.singh, vijjinianvesh.rao}@research.iiit.ac.in m.shrivastava@iiit.ac.in Abstract In this paper, we leverage social media platforms such as twitter for developing corpus across multiple languages. The corpus creation methodology is applicable for resource- scarce languages provided the speakers of that particular language are active users on social media platforms. We present an approach to extract social media microblogs such as tweets (Twitter). In this paper, we create corpus for multilingual sentiment analysis and emoji prediction in Hindi, Bengali and Telugu. Further, we perform and analyze multiple NLP tasks utilizing the corpus to get interesting observations. 1 Introduction Twitter has become a valuable source of data for various NLP studies such as sentiment analysis, polarity detection and emoji prediction. Although, copious amount of research studies have been conducted on twitter data, a majority of them deal with English tweets. This is due to several factors. First, English dominates in the mix of languages on Twitter. According to (Hong et al., 2011), more than 50% of tweets are in English. Outside of the largest five Twitter languages (given in Figure 1), other languages represent just under 1% of Twitter traffic each. We primarily focus on collecting data for sentiment analysis. Additionally, taking the premise that social media devices like emojis convey sentiment of their respective tweet, we provide a methodology for collecting tweets with emojis for any language irrespective of its resource availability. Going beyond sentiment analysis, we also collect data without any preconditions on presence or absence of emojis in the tweets which could be used to draw interesting social media analytics both within and across languages or linguistic communities. We discuss, specifically, about resource-poor languages because such discourse is available on resource-rich languages like English and Spanish, whereas resource-poor languages are largely ig- nored. For such resource-poor, but yet widely spoken, languages (especially in multilingual com- munities that have their predominant literature in a language other than their native tongue), we observe social media as a good data source. In this paper, we look at Twitter to collect data in Telugu, Hindi and Bengali, predominantly spoken in the Indian subcontinent. In these re- gions, English is the language of administration and is increasingly becoming the lingua franca. This, also, explains the lack of sizable corpora despite the relatively large number of speakers. 2 Related Work Sharing of corpora or resources is important for researchers to compare results with each other, pushing the boundaries of the state-of-the-art model. Openly available corpora reduce the efforts of researchers in constructing the same corpus others developed merely for comparison. However, Twitter’s terms of service, under which sharing of aggregated resources (tweets) is barred, prove an unnecessary obstacle in this regard. For example, the Edinburgh Twitter corpus (Petrovic et This work is licensed under a Creative Commons Attribution 4.0 International License. …

Content truncated to 500 words