Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data

Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Nada AlMarwani, Mohamed Al-Badrashiny Department of Computer Science The George Washington University {mtdiab, mghoneim, abhawwari, fghamdi, nadaoh, badrashiny}@gwu.edu Abstract We present our effort to create a large Multi-Layered representational repository of Linguistic Code-Switched Arabic data. The process involves developing clear annotation standards and Guidelines, streamlining the annotation process, and implementing quality control measures. We used two main protocols for annotation: in-lab gold annotations and crowd sourcing annotations. We developed a web-based annotation tool to facilitate the management of the annotation process. The current version of the repository contains a total of 886,252 tokens that are tagged into one of sixteen code-switching tags. The data exhibits code switching between Modern Standard Arabic and Egyptian Dialectal Arabic representing three data genres: Tweets, commentaries, and discussion fora. The overall Inter-Annotator Agreement is 93.1%. Keywords: Code Switching, Computational Linguistic Resources, Annotation, Sociolinguistics. 1. Introduction Linguistic Code Switching (CS) is a common practice among multilingual speakers in which they switch between their common languages in written and spoken communication. A Spanish-English blog entry illustrates this: “She told me that mi esposo looks like un buen hombre.” (“She told me that my husband looks like a good man”). CS is typically present on the inter-sentential, intra-sentential (mixing of words from multiple languages in the same utterance) and even morphological (mixing of morphemes) levels. This phenomenon can be observed in different linguistic levels of representation for different language pairs: phonological, morphological, lexical, syntactic, semantic, and discourse/pragmatic switching. For multilingual speakers, CS is pervasive in their spoken and in informal written genres such as email and web blogs. CS presents serious challenges for language technologies, including parsing, machine translation (MT), automatic speech recognition (ASR), semantic processing, and information retrieval (IR) and extraction (IE). Techniques trained for one language quickly break down when there is input from another. Being able to predict/identify probable switch points, as well as which dialect/language a speaker is switching to, enables applications to adapt their models. A major barrier to research on CS has been the lack of large, consistently and accurately annotated corpora of CS data. In the shared task for “Language Identification in Code-Switched Data” (Solorio et al., 2014), the first set of annotated data was created which focused on social media and covered four language pairs: Modern Standard Arabic - Dialectal Arabic (MSA-DA), Mandarin - English (MAN-EN), Nepali - English (NEP-EN), and Spanish - English (SPA-EN). In this work we present our effort to build a large repository of CS data that will cover multiple language pairs and dialects. We started by focusing on Arabic Language. Arabic is a Semitic language spoken by over 300M people worldwide. CS between MSA and DA is widespread among native speakers of Arabic. MSA is the language of education used in formal speeches and settings, while DA is the everyday spoken variant; even minimally educated Arabic speakers speak two languages. While there is considerable lexical …

Content truncated to 500 words