Identification of Parallel Text Pairs Using Fingerprints

International Conference RANLP 2009 - Borovets, Bulgaria, pages 135–138 Identification of Parallel Text Pairs Using Fingerprints Martin Hassel † † DSV, KTH - Stockholm University Forum 100 164 40 Kista, Sweden xmartin@dsv.su.se Hercules Dalianis †‡ ‡ Euroling AB, SiteSeeker Igeldammsgatan 22c 112 49 Stockholm, Sweden hercules@dsv.su.se Abstract When creating dictionaries for use in for example cross- language search engines, one often uses a word alignment system that takes parallel or comparable text pairs as input and produces a word list. Multilingual web sites may contain parallel texts but these can be difficult to detect. In this article we describe an experiment on automatic identification of parallel text pairs. We utilize the frequency distribution of word initial letters in order to map a text in one language to a corresponding text in another in the JRC-Acquis corpus (European Council legal texts). Using English and Swedish as language pair, and running a ten-fold random pairing, the algorithm made 87 percent correct matches (baseline-random 50 percent). Attempting to map the correct text among nine randomly chosen false matches and one true yielded a success rate of 68 percent (baseline-random 10 percent). Keywords Cross Language Information Retrieval, Identification of Parallel Text, Prefix Frequency Distribution, A-priori Probability. 1. Introduction Dictionaries are an important part of natural language processing tasks and linguistic work. Domain-specific dictionaries can for example be used in cross-language web and intranet search engines. Word alignment tools are often used for the creation of bilingual word lists. These tools need parallel corpora to work properly. One source is Internet and the multilingual web sites there. Unfortunately these web sites are often only parallel with regard to web pages. In [6] and in [2] are described different heuristics to download and identify parallel text. However, these methods are not enough since the downloaded parallel text still can be very noisy. For example [13] found only 45 percent parallel text pairs on the multilingual parallel web site Hallå Norden (Hello Scandinavia) that was intended to be completely parallel and the parallel pages contained 5 percent non- parallel elements. Therefore, we found a need to develop and evaluate a new method for identifying parallel and non-parallel texts in corpora covering different language pairs. 2. Related Work The distinction between a parallel and a comparable corpus is very important and has been discussed in for example [10] and also in [3]. Freely available multilingual resources are often noisy and non-parallel sections need to be removed. Many methods for identifying such sections automatically have been proposed. Maximum entropy (ME) classification is used in [7] in order to improve machine translation performance. From large Chinese, Arabic and English non- parallel newspaper corpora, parallel data was extracted. For this method, a bilingual dictionary and a small amount of parallel data for the ME classifier is needed. By selecting pairs of similar documents from two monolingual corpora, all possible sentence pairs are passed through a word- overlap based filter and then sent to the ME classifier. The authors reported significant improvements …

Content truncated to 500 words