Hybrid Parallel Sentence Mining from Comparable Corpora

Hybrid Parallel Sentence Mining from Comparable Corpora Dan Ștefănescu RACAI Calea 13 Septembrie, 13 Bucharest, Romania danstef@racai.ro Radu Ion RACAI Calea 13 Septembrie, 13 Bucharest, Romania radu@racai.ro Sabine Hunsicker DFKI Stuhlsatzenhausweg 3, 66123 Saarbrücken, Germany sabine.hunsicker@dfki.de Abstract This paper presents a fast and accurate parallel sentence mining algorithm for comparable corpora called LEXACC based on the Cross-Language Infor- mation Retrieval framework combined with a trainable translation similarity measure that detects pairs of parallel and quasi-parallel sentences. LEXACC ob- tains state-of-the-art results in compari- son with established approaches. 1 Introduction Mining for parallel sentences in comparable cor- pora is much more difficult than aligning sen- tences in parallel corpora. Sentence alignment in parallel corpora usually exploits simple empirical evidence (turned into assumptions) such as (i) the length of a sentence is proportional with the length of its translation and (ii) the discourse flow is necessarily the same in both parts of the bi-text (Gale and Church, 1993). Thus, the ex- traction tools search for parallel sentences around the same (relative) text positions, making sen- tence alignment a much easier task when com- pared to kind of work undertaken here. For comparable corpora, the second assump- tion does not hold. Parallel sentences, should they exist at all, are scattered all around the source and target documents, and so, any two sentences1 have to be processed in order to de- termine if they are parallel or not. Also, we aim at finding pairs of quasi-parallel sentences that are not entirely parallel but contain spans of con- tiguous text that is parallel. Thus, finding parallel sentences in comparable corpora is confronted © 2012 European Association for Machine Translation. 1 Or a carefully selected set of sentence pairs as we will see in the next sections. with the vast search space one has to consider since any positional clues indicating parallel or partially parallel sentences are not available. The brute force approach is to analyze every element of the Cartesian product between the two sets containing sentences in the source and target languages. This approach is clearly impractical since the resulting algorithm would be very slow and/or memory consuming. 2 To reduce the search space, we turned to a framework that be- longs to Information Retrieval: Cross-Language Information Retrieval (CLIR). The idea is simple: use a search engine to find sentences in the target corpus that are the most probable translations of a given sentence from the source corpus. The first step is to consider the target sentences as documents and index them. Then, for each sen- tence in the source corpus, one selects the con- tent words and translates them into the target language according to a given dictionary. The translations are used to form a Boolean query which is then fed to the search engine. The top hits are considered to be translation candidates. Using the CLIR approach to select a set of candidate target sentences (out of all target sen- tences) for the input source sentence is one way to dramatically reduce …

Content truncated to 500 words