An Arabizi-English social media statistical machine translation system

An Arabizi-English Social Media Statistical Machine Translation System Jonathan May∗ jonmay@isi.edu USC Information Sciences Institute, Marina del Rey, CA 90292 Yassine Benjira ybenjira@sdl.com Abdessamad Echihabi aechihabi@sdl.com SDL Language Weaver, Los Angeles, CA 90045 Abstract We present a machine translation engine that can translate romanized Arabic, often known as Arabizi, into English. With such a system we can, for the first time, translate the mas- sive amounts of Arabizi that are generated every day in the social media sphere but until now have been uninterpretable by automated means. We accomplish our task by leveraging a ma- chine translation system trained on non-Arabizi social media data and a weighted finite-state transducer-based Arabizi-to-Arabic conversion module, equipped with an Arabic character- based n-gram language model. The resulting system allows high capacity on-the-fly translation from Arabizi to English. We demonstrate via several experiments that our performance is quite close to the theoretical maximum attained by perfect deromanization of Arabizi input. This constitutes the first presentation of a high capacity end-to-end social media Arabizi-to-English translation system. 1 Introduction Arabic-English machine translation systems generally expect Arabic input to be rendered as Arabic characters. However, a substantial amount of Arabic in the wild is rendered in Latin characters, using an informal mapping known as Romanized Arabic, Arabish, or Arabizi. Ara- bizi mainly differs from strict transliteration or romanization schemes such as that of Buckwalter or ALA-LC1 in that it is not standardized. Usage is inconsistent and varies between different dialect groups and even individuals. Despite these drawbacks, Arabizi is widely used in social media contexts such as Twitter. As can be seen in Figure 1, it is not uncommon for users to use a mix of Arabic script, Arabizi, and even foreign languages such as English in their daily stream of communication. Arabizi can be viewed as a romanization of Arabic consisting of both transliteration and transcription mappings. Transliteration is the act of converting between orthographies in a way that preserves the character sequence of the original orthography. An example of transliteration in Arabizi is the mapping of the character � to ‘3’ due to the similarity of the glyphs. Tran- scription (specifically, phonetic transcription) between orthographies is the act of converting in a way that preserves the spoken form of the original orthography as interpreted by a reader of the new orthography’s presumed underlying language. An example of transcription in Arabizi is the mapping of the character �� to any of ‘g’, ‘j’, or “dj.” This reflects the fact that in various ∗ This work was done while the first author was employed by SDL Language Weaver 1http://www.loc.gov/catdir/cpso/romanization/arabic.pdf Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 329 Figure 1: Examples of Arabizi mixed with Arabic and English in Twitter dialects �� may be pronounced as [g] (as in god), [Z] (as in vision), or [> dZ] (as in juice), and that the digraph “dj” is used in French for [> dZ]. For a machine translation system to properly …

Content truncated to 500 words