Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification

Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification Wajdi Zaghouani, 1 Anis Charfi2 1 College of Humanities and Social Sciences, Hamad Bin Khalifa University, Qatar 2 Information Systems Program, Carnegie Mellon University Qatar E-Mail: wzaghouani@hbku.edu.qa, acharfi@qatar.cmu.edu Abstract In this paper, we present Arap-Tweet, which is a large-scale and multi-dialectal corpus of Tweets from 11 regions and 16 countries in the Arab world representing the major Arabic dialectal varieties. To build this corpus, we collected data from Twitter and we provided a team of experienced annotators with annotation guidelines that they used to annotate the corpus for age categories, gender, and dialectal variety. During the data collection effort, we based our search on distinctive keywords that are specific to the different Arabic dialects and we also validated the location using Twitter API. In this paper, we report on the corpus data collection and annotation efforts. We also present some issues that we encountered during these phases. Then, we present the results of the evaluation performed to ensure the consistency of the annotation. The provided corpus will enrich the limited set of available language resources for Arabic and will be an invaluable enabler for developing author profiling tools and NLP tools for Arabic. Keywords: Multi-dialectal corpus, Author Profiling, Social media 1. Introduction Arabic is a challenging language when it comes to building Natural Language Processing tools and applications. In fact, the complexity of Arabic is present at the various levels of linguistic representation (phonology, orthography, morphology, and syntax). Even though there were some advances in the field of Arabic Natural Language Processing, the Arabic language is still lagging behind other languages such as English in terms of availability of the required resources to address author profiling (Rosso et al., 2018; Zaghouani, 2014). In fact, a person’s language use reveals much about their profile. However, research on author profiling has always been constrained by the limited availability of training data, since collecting textual data with the appropriate meta-data requires a significant collection and annotation effort. For every text, the characteristics of the author have to be known in order to successfully profile the author. For the Arabic language, to the best of our knowledge, there is no corpus freely available for the detection of age, gender and dialectal variety. Most of the existing corpora are available for English or other European languages (Celli et al., 2013). Having a large amount of data remains the key to achieving reliable results in the task of author profiling. This paper presents the work carried out within the framework of the Arabic Author Profiling Project (ARAP),1 a research project funded by Qatar National Research Fund. This project aims at developing author profiling resources and tools for the Arabic language and using them in the context of cyber-security. More specifically, author profiling in the context of ARAP could be useful for forensic investigations to narrow the set of potential authors when receiving a threat message. While a few research efforts on author profiling …

Content truncated to 500 words