Creation of comparable corpora for English-Urdu, Arabic, Persian

Creation of Comparable Corpora for English-{Urdu, Arabic, Persian} Murad Abouammoh, Kashif Shah*, Ahmet Aker* King Saud University, KSA, *The University of Sheffield, UK E-mail: muabouammoh@ksu.edu.sa, kashif.shah@sheffield.ac.uk, ahmet.aker@sheffield.ac.uk Abstract Statistical Machine Translation (SMT) relies on the availability of rich parallel corpora. However, in the case of under-resourced languages or some specific domains, parallel corpora are not readily available. This leads to under-performing machine translation systems in those sparse data settings. To overcome the low availability of parallel resources the machine translation community has recognized the potential of using comparable resources as training data. However, most efforts have been related to European languages and less in middle-east languages. In this study, we report comparable corpora created from news articles for the pair English –{Arabic, Persian, Urdu} languages. The data has been collected over a period of a year, entails Arabic, Persian and Urdu languages. Furthermore using the English as a pivot language, comparable corpora that involve more than one language can be created, e.g. English- Arabic - Persian, English - Arabic - Urdu, English – Urdu - Persian, etc. Upon request the data can be provided for research purposes. Keywords: Comparable Corpora for Arabic, Urdu, Persian and English 1. Paper Statistical Machine Translation (SMT) relies heavily on the quality and quantity of bilingual parallel corpora (Brown et al., 1993, Och and Ney, 2002, Koehn, 2010). Most known parallel corpora are collected from the United Nation or the European parliament and are mostly about the legal domain (Koehn, 2005). However, often parallel resources are not readily available for under- resourced languages or other specific narrow domains. This leads to under-performing machine translation systems in those sparse data settings. To overcome the low availability of parallel resources the machine translation community has recognized the potential of using comparable resources as training data (Rapp, 1999, Munteanu and Marcu, 2002, Sharoff et al., 2006, Munteanu and Marcu, 2006, Kumano et al., 2007, Kauchak and Barzilay, 2006, Callison-Burch et al., 2006, Barzilay and McKeown, 2001, Nakov, 2008, Zhao et al., 2008, Marton et al., 2009, Aker et al., 2012) Without a doubt the web is the largest source that can be used to gather comparable corpora (Resnik and Smith, 2003, Huang et al., 2010). Newswire have been also explored to gather comparable corpora. The ACCURAT1 project, for instance, collected comparable corpora for 12 different European languages using the web, through Wikipedia and News articles (Skadiņa et al., 2012a, Skadiņa et al., 2012b). A follow up project, TaaS2, expanded this idea to all EU languages. In both projects it has been shown that comparable corpora have positive impact on SMT as well as in creation bilingual terminology resources. Apart from those mentioned studies and projects there have been many workshops focusing solely on building and using comparable corpora (BUCC3). The amount and diverse studies and efforts show that 1 http://www.accurat-project.eu/ 2 http://www.taas-project.eu/ 3 https://comparable.limsi.fr/ comparable corpora is certainly a useful resource to determine helpful material for SMT but also for related studies such as bilingual term extraction, cross-lingual …

Content truncated to 500 words