Extracting an English-Persian Parallel Corpus from Comparable Corpora

Extracting an English-Persian Parallel Corpus from Comparable Corpora Akbar Karimi*, Ebrahim Ansari*, Bahram Sadeghi Bigham* * Department of Computer Sciences and Information Technology, Institute for Advanced Studies in Basic Sciences (IASBS), Zanjan, Iran {ak.karimi, ansari, b_sadeghi_b}@iasbs.ac.ir Abstract Parallel data are an important part of a reliable Statistical Machine Translation (SMT) system. The more of these data are available, the better the quality of the SMT system. However, for some language pairs such as Persian-English, parallel sources of this kind are scarce. In this paper, a bidirectional method is proposed to extract parallel sentences from English and Persian document aligned Wikipedia. Two machine translation systems are employed to translate from Persian to English and the reverse after which an IR system is used to measure the similarity of the translated sentences. Adding the extracted sentences to the training data of the existing SMT systems is shown to improve the quality of the translation. Furthermore, the proposed method slightly outperforms the one-directional approach. The extracted corpus consists of about 200,000 sentences which have been sorted by their degree of similarity calculated by the IR system and is freely available for public access on the Web1. Keywords: Parallel Sentence Extraction, Comparable Corpora, Statistical Machine Translation, Wikipedia, English-Persian Corpus 1. Introduction Due to the abundance of data on the Internet, statistical machine translation (SMT) has gained more popularity. In order to build an SMT system, parallel corpora are of high importance. These parallel resources which have been aligned on the sentence level in two languages (source and target), are used in the training phase of the SMT system. Therefore, the larger the parallel corpora are, the better the performance of the SMT system is. However, for some language pairs such as Persian-English not much data of this type is available. This lack of parallel data has led researchers to make use of other available data called comparable corpora which contain a mixture of parallel and partially parallel sentences. They can be given a certain degree of comparability which ranges from lowly comparable to highly comparable (Li and Gaussier, 2010). Research shows that using these corpora can help improve the performance of the SMT system. There are several sources such as news articles, company manuals, Wikipedia articles, and so forth which can be considered as comparable corpora. In this work, our aim is to extract parallel sentences for Persian-English language pair from Wikipedia documents using a new approach to improve the Persian-English SMT system. Our method consists of two main parts: translation and information retrieval. For the translation part (Persian to English and English to Persian), we employed Moses Toolkit developed by Koehn et al., (2007) which is an open-source toolkit developed for phrase-based translation and for the IR step we utilized the Lucene IR system2. Lucene has been designed to work with queries which are fed into the system one by one and the results shown by the IR system for a query are numbers representing the degree of the documents’ relevance to the …

Content truncated to 500 words