An Automatic Learning of an Algerian Dialect Lexicon by using Multilingual Word Embeddings

An Automatic Learning of an Algerian Dialect Lexicon by using Multilingual Word Embeddings Karima Abidi, Kamel Sma¨ıli SMarT Group, LORIA, F-54600, France {karima.abidi, kamel.smaili}@loria.fr Abstract The goal of this work consists in building automatically from a social network (Youtube) an Algerian dialect lexicon. Each entry of this lexicon is composed by a word, written in Arabic script (modern standard Arabic or dialect) or Latin script (Arabizi, French or English). To each word, several transliterations are proposed, written in a script different from the one used for the word itself. To do that, we harvested and aligned an Algerian dialect corpus by using an iterative method based on multlingual word embeddings representation. The multlinguality in the corpus is due to the fact that Algerian people use several languages to post comments in social networks: Modern Standard Arabic (MSA), Algerian dialect, French and sometimes English. In addition, the users of social networks write freely without any regard to the grammar of these languages. We tested the proposed method on a test lexicon, it leads to a score of 73% in terms of F-measure. Keywords: Multilingual word embeddings, Algerian dialect, CBOW, comparability 1. Introduction The wide use of social networks arises several new NLP is- sues: stretched letters, misspelled words, use of emoticons, condensed writing, etc. For the use of Arabic in social networks, the same phenomena are observed, henceforth other issues are also noticed and especially for Arabic dialects. In this work, we are interested by the Algerian Arabic dialect. One needs to know that in Algeria, people speak their mother tongue, which is an Arabic dialect, but could speak the official language that is Modern Standard Arabic (MSA), French and sometimes English. Nevertheless, what makes the issue more challenging is that people can mix in the same sentence the whole previous languages. Since this vernacular language is not written and no standardization exists, people write their comments in a free way. They do not pay special attention to grammar, consequently for uneducated people, they write a word as they want or, at the best, such as it is pronounced. People write sometimes Algerian dialect by using the Latin script (LS) because they are influenced by the French culture, this phenomenon is named in the community working on dialect: Arabizi. This could be explained also by the fact that in Algeria the mobile phone keyboards are in French that makes writing in Arabic more difficult. This obviously constitutes a serious issue because Arabic NLP tools could not be used on Latin script and more especially because the corresponding resources are not available. Unfortunately, the NLP tools in French cannot be used either since the written words in LS are not necessarily in French. In the following, we give an example of a post extracted from YouTube written in LS: {ouiii 7na haka kima galha bn chnt w li maybghounach w ha l7itan ha l7itan}. That should be written in Arabic script (AS) such as: { �������� �� �������� �� � ��������������� �� …

Content truncated to 500 words