Mapping Rules for Building a Tunisian Dialect Lexicon and Generating Corpora

International Joint Conference on Natural Language Processing, pages 419–428, Nagoya, Japan, 14-18 October 2013. Mapping Rules for Building a Tunisian Dialect Lexicon and Generating Corpora Rahma Boujelbane ANLP Research Group, MIRACL Lab, University of Sfax, Tunisia rahma.boujelbane@gmail.com Meriem Ellouze Khemekhem ANLP Research Group, MIRACL Lab, University of Sfax, Tunisia mariem.ellouze@planet.com Lamia Hadrich Belguith ANLP Research Group, MIRACL Lab, University of Sfax, Tunisia l.belguith@fsegs.rnu.tn Abstract Nowadays in tunisia, the arabic Tunisian Dialect (TD) has become progressively used in interviews, news and debate programs instead of Modern Standard Arabic (MSA). Thus, this gave birth to a new kind of language. Indeed, the majority of speech is no longer made in MSA but alternates between MSA and TD. This situation has important negative consequences on Automatic Speech Recognition (ASR): since the spoken dialects are not officially written and do not have a standard orthography, it is very costly to obtain adequate annotated corpora to use for training language models and building vocabulary. There are neither parallel corpora involving Tunisian dialect and MSA nor dictionaries. In this paper, we describe a method for building a bilingual dictionary using explicit knowledge about the relation between TD and MSA. We also present an automatic process for creating Tunisian Dialect (TD) corpora. 1 Introduction Recently, due to the changes that have occurred in the Arab world, we noticed a new remarkable diversity in the media. The Arabic dialects used in daily life have become progressively used and represented in interviews, news and debate programs instead of Modern Standard Arabic (MSA). In Tunisia, for example, the revolution has affected not only the people but also the media. For that reason, the media programs have been changed: television channels, political debates and broadcasts news have been multiplied. This gave birth to a new kind of language. Indeed, the majority of speech is no longer made in MSA but alternates between MSA and Tunisian Dialect (TD). Thus, we can distinguish in the same speech, MSA words, TD words and MSA-TD words such as a word with an MSA component (root) and dialectal affixes. This situation poses significant challenges to NLP: In fact, applying NLP tools designed for MSA to TD yields a significantly lower performance, making it imperative to direct research towards building resources and tools that make it possible to process this kind of language. In our case, we aim to convert this new language into text. However, this process presents a series of linguistic and computational challenges. Some of these relate to language modeling: studying large amounts of text to learn about patterns of words in a language. This task is complicated because of the total lack of TD resources, whether parallel TD-MSA text or dictionaries. In this paper, we describe a method that helps to create Tunisian Dialect (TD) text corpora and the associated lexical resources and also build a bilingual MSA-TD dictionary. This paper is organized as follows: After discussing related work, we present our method to deal with the lack of Tunisian resources (Section3). We …

Content truncated to 500 words