A Minimally Supervised Approach for Detecting and Ranking Document Translation Pairs

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 207–216, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics A Minimally Supervised Approach for Detecting and Ranking Document Translation Pairs Kriste Krstovski David A. Smith Department of Computer Science Department of Computer Science University of Massachusetts Amherst University of Massachusetts Amherst Amherst, MA 01003, USA Amherst, MA 01003, USA kriste@cs.umass.edu dasmith@cs.umass.edu Abstract We describe an approach for generating a ranked list of candidate document transla- tion pairs without the use of bilingual dic- tionary or machine translation system. We developed this approach as an initial, filter- ing step, for extracting parallel text from large, multilingual—but non-parallel— corpora. We represent bilingual documents in a vector space whose basis vectors are the overlapping tokens found in both lan- guages of the collection. Using this repre- sentation, weighted by tf·idf, we compute cosine document similarity to create a ranked list of candidate document transla- tion pairs. Unlike cross-language informa- tion retrieval, where a ranked list in the target language is evaluated for each source query, we are interested in, and evaluate, the more difficult task of finding translated document pairs. We first perform a feasi- bility study of our approach on parallel col- lections in multiple languages, representing multiple language families and scripts. The approach is then applied to a large bilingual collection of around 800k books. To avoid the computational cost of ) ( On2 document pair comparisons, we employ locality sen- sitive hashing (LSH) approximation algo- rithm for cosine similarity, which reduces our time complexity to (log) n On . 1 Introduction A dearth of parallel data has been, and still is, a major problem for developing highly reliable sta- tistical machine translation systems in many lan- guages and domains. There have been many proposed approaches for alleviating this problem by utilizing techniques for creating and extracting parallel documents, sentences or phrases from comparable bilingual data available on the open web (Resnik and Smith, 2003), such as Wikipedia articles (Smith et. al, 2010), to name a few, or through digitized archives from various sources (Zhao and Vogel, 2002), (Munteanu and Marcu, 2005). In general, in the process of utilizing comparable corpora to obtain sentence-aligned bilingual text, the first step involves performing initial filtering where text entities from both language collections are compared to each other and based on compari- son score they are matched and grouped as poten- tial translation candidate pairs. After this initial step, text entity pairs or tuples are further analyzed in order to extract parallel sentence pairs. In this paper we only focus on this initial step. We present a novel exploration of approaches that retrieve ac- tual document translation pairs without the use of any bilingual resources such as lexicons or sen- tence aligned bitext. Rather than solving separate retrieval or translation problems for each source language document, we retrieve translation pairs from the space of all pos- sible bilingual document pairs. Most machine 207 translation (MT) and information retrieval (IR) systems rely …

Content truncated to 500 words