From Words to Corpora: Recognizing Translation

From Words to Corpora: Recognizing Translation Noah A. Smith Department of Computer Science Johns Hopkins University Baltimore, MD 21218 USA nasmith@cs.jhu.edu Abstract This paper presents a technique for discover- ing translationally equivalent texts. It is com- prised of the application of a matching algo- rithm at two different levels of analysis and a well-founded similarity score. This approach can be applied to any multilingual corpus us- ing any kind of translation lexicon; it is there- fore adaptable to varying levels of multilingual resource availability. Experimental results are shown on two tasks: a search for matching thirty-word segments in a corpus where some segments are mutual translations, and classifi- cation of candidate pairs of web pages that may or may not be translations of each other. The latter results compare competitively with pre- vious, document-structure-based approaches to the same problem. 1 Introduction As in most areas of natural language process- ing, recent approaches to machine translation have turned increasingly to statistical modeling of the phenomenon (translation models) (Berger et al., 1994). Such models are learned auto- matically from data, typically parallel corpora: texts in two or more languages that are mutual translations. As computational resources have become more powerful and less expensive, the task of training translation models has become feasible (Al-Onaizan et al., 1999), as has the task of translating (or “decoding”) text using such models (Germann et al., 2001). However, the success of the statistical approach to trans- lation (and also to other multilingual applica- tions that utilize parallel text) hangs crucially on the quality, quantity, and diversity of data used in parameter estimation. If translation is a generative process, then one might consider its reverse process of recognition: Given two documents, might it be determined fully automatically whether they are transla- tions of each other? The ability to detect translations of a doc- ument has numerous applications. The most obvious is as a means to build a parallel corpus from a set of multilingual documents that con- tains some translation pairs. Examples include mining the World-Wide Web for parallel text (Resnik, 1999; Nie et al., 1999; Ma and Liber- man, 1999) and building parallel corpora from comparable corpora such as multilingual collec- tions of news reports. Another use of trans- lation detection might be as an aid in align- ment tasks at any level. For example, consider the task of aligning NP chunks (and perhaps also the extra-NP material) in an NP-bracketed parallel corpus; a chunk-level similarity score (Fluhr et al., 2000) built from a word-level model could be incorporated into a framework that involves bootstrapping more complex mod- els of translation from simpler ones (Berger et al., 1994). Finally, reliable cross-lingual dupli- cate detection might improve performance in n- best multilingual information retrieval systems; at the same time, by detecting the existence of a translation in a multilingual corpus, the cost of translating a document of interest is eliminated. I present here an algorithm for classifying document pairs as either translationally equiv- alent or not, which can …

Content truncated to 500 words