Large Scale Parallel Document Mining for Machine Translation

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1101–1109, Beijing, August 2010 Large Scale Parallel Document Mining for Machine Translation Jakob Uszkoreit Jay M. Ponte Ashok C. Popat Moshe Dubiner Google, Inc. {uszkoreit,ponte,popat,moshe}@google.com Abstract A distributed system is described that re- liably mines parallel text from large cor- pora. The approach can be regarded as cross-language near-duplicate detec- tion, enabled by an initial, low-quality batch translation. In contrast to other ap- proaches which require specialized meta- data, the system uses only the textual con- tent of the documents. Results are pre- sented for a corpus of over two billion web pages and for a large collection of digi- tized public-domain books. 1 Introduction While the World Wide Web provides an abun- dance of readily available monolingual text, par- allel data is still a comparatively scarce resource, yet plays a crucially important role in training sta- tistical machine translation systems. We describe an approach to mining document- aligned parallel text to be used as training data for a statistical machine translation system. Pre- vious approaches have focused on rather homo- geneous corpora and relied on metadata such as publication dates (Munteanu and Marcu, 2005; Munteanu and Marcu, 2006; Udupa et al., 2009; Do et al., 2009; Abdul-Rauf and Schwenk, 2009) or information about document structure (Resnik and Smith, 2003; Chen and Nie, 2000). In large and unstructured collections of documents such as the Web, however, metadata is often sparse or un- reliable. Our approach, in contrast, scales com- putationally to very large and diverse collections of documents and does not require metadata. It is based solely on the textual contents of the input documents. Casting the problem as one of cross-language near duplicate detection, we use a baseline ma- chine translation system to translate all input doc- uments into a single language. However, the words and phrases that are most discriminatory for the purposes of information retrieval and du- plicate detection are the relatively rare ones, pre- cisely those that are less likely to be translated well by the baseline translation system. Our approach to circumvent this problem and to avoid the prohibitive quadratic computational complexity of the naive approach of performing a comparison of every possible pair of input docu- ments is similar to previous work in near duplicate detection (Broder, 2000; Henzinger, 2006; Man- ber, 1994) and noisy data retrieval (Harding et al., 1997). We use shingles consisting of word n-grams to construct relatively rare features from more com- mon, in-vocabulary words. For each input doc- ument, we identify a comparatively small set of candidate pairings with documents sharing at least a certain number of such features. We then per- form a more expensive comparison between each document and all documents in its candidate set using lower order n-gram features that would typ- ically be too frequent to be used efficiently in forming candidate pairings, but provide a higher coverage of the scored document pairs. Another important aspect of our approach is that it can …

Content truncated to 500 words