Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover’s Distance

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 616–625 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 616 Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover’s Distance Ahmed El-Kishky Facebook AI ahelk@fb.com Francisco Guzm´an Facebook AI fguzman@fb.com Abstract Document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. Such aligned data can be used for a va- riety of NLP tasks from training cross-lingual representations to mining parallel data for ma- chine translation. In this paper we develop an unsupervised scoring function that leverages cross-lingual sentence embeddings to compute the semantic distance between documents in different languages. These semantic distances are then used to guide a document alignment algorithm to properly pair cross-lingual web documents across a variety of low, mid, and high-resource language pairs. Recognizing that our proposed scoring function and other state of the art methods are computationally intractable for long web documents, we uti- lize a more tractable greedy algorithm that per- forms comparably. We experimentally demon- strate that our distance metric performs better alignment than current baselines outperform- ing them by 7% on high-resource language pairs, 15% on mid-resource language pairs, and 22% on low-resource language pairs. 1 Introduction While the Web provides a large amount of mono- lingual text, cross-lingual parallel data is more difficult to obtain. Despite its scarcity, parallel cross-lingual data plays a crucial role in a variety of tasks in natural language processing such as machine translation. Previous works have shown that training on sentences extracted from parallel or comparable documents mined from the Web can improve machine translation models (Munteanu and Marcu, 2005) or learning word-level transla- tion lexicons (Fung and Yee, 1998; Rapp, 1999). Other tasks that leverage these parallel texts include cross-lingual information retrieval, document clas- sification, and multilingual representations such as Source Target Figure 1: Documents in a source and target langauge in the same web-domain. Solid lines indicate cross- lingual document pairs. XLM (Lample and Conneau, 2019). Document alignment is a method for obtaining cross-lingual parallel data that seeks to pair documents in dif- ferent languages such that pairs are translations or near translations of each other. As seen in Figure 1, this involves a one-to-one pairing of documents in a source language with documents in a target language. To automate and scale the process of identifying these documents pairs, we introduce an approach to accurately mine comparable web documents across a variety of low, mid, and high-resource language directions. Previous approaches have been applied to homogeneous corpora, however mining the Web involves analyzing a variety of heterogeneous data sources (Koehn et al., 2002). Other approaches rely on corpus-specific features such as metadata and publication date which can be inconsistent and un- reliable (Munteanu and Marcu, 2005; AbduI-Rauf and Schwenk, 2009). Related methods utilize docu- ment structure when calculating document similar- ity (Resnik and Smith, 2003; …

Content truncated to 500 words