Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197–3203 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3197 Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings Mikel Artetxe University of the Basque Country (UPV/EHU)∗ mikel.artetxe@ehu.eus Holger Schwenk Facebook AI Research schwenk@fb.com Abstract Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collect- ing and filtering large parallel corpora. In this paper, we propose a new method for this task based on multilingual sentence embed- dings. In contrast to previous approaches, which rely on nearest neighbor retrieval with a hard threshold over cosine similarity, our proposed method accounts for the scale in- consistencies of this measure, considering the margin between a given sentence pair and its closest candidates instead. Our experiments show large improvements over existing meth- ods. We outperform the best published results on the BUCC mining task and the UN recon- struction task by more than 10 F1 and 30 preci- sion points, respectively. Filtering the English- German ParaCrawl corpus with our approach, we obtain 31.2 BLEU points on newstest2014, an improvement of more than one point over the best official filtered version. 1 Introduction While Neural Machine Translation (NTM) has obtained breakthrough improvements in standard benchmarks, it is known to be particularly sen- sitive to the size and quality of the training data (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018). In this context, effective ap- proaches to mine and filter parallel corpora are crucial to apply NMT in practical settings. Traditional parallel corpus mining has relied on heavily engineered systems. Early approaches were mostly based on metadata information from web crawls (Resnik, 1999; Shi et al., 2006). More recent methods focus on the textual content in- stead. For instance, Zipporah learns a classifier ∗This work was performed during an internship at Face- book AI Research. over bag-of-word features to distinguish between ground truth translations and synthetic noisy ones (Xu and Koehn, 2017). STACC uses seed lexical translations induced from IBM alignments, which are combined with set expansion operations to score translation candidates through the Jaccard similarity coefficient (Etchegoyhen and Azpeitia, 2016; Azpeitia et al., 2017, 2018). Many of these approaches rely on cross-lingual document re- trieval (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005, 2006; Abdul-Rauf and Schwenk, 2009) or machine translation (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018). More recently, a new research line has shown promising results using multilingual sentence em- beddings alone1 (Schwenk, 2018; Guo et al., 2018). These methods use an NMT inspired encoder-decoder to train sentence embeddings on existing parallel data, which are then directly ap- plied to retrieve and filter new parallel sentences using nearest neighbor retrieval over cosine simi- larity with a hard threshold (Espa˜na-Bonet et al., 2017; Hassan et al., 2018; Schwenk, 2018). In this paper, we argue that this retrieval method suffers from the scale of cosine similarity not be- ing globally consistent. As …

Content truncated to 500 words