Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 165–176 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64017 Effective Parallel Corpus Mining using Bilingual Sentence Embeddings Mandy Guoa∗, Qinlan Shenb†∗, Yinfei Yanga∗, Heming Gea, Daniel Cera, Gustavo Hernandez Abregoa, Keith Stevensa, Noah Constanta, Yun-Hsuan Sunga, Brian Stropea, Ray Kurzweila aGoogle AI Mountain View, CA, USA bCarnegie Mellon University Pittsburgh, PA, USA Abstract This paper presents an effective approach for parallel corpus mining using bilingual sen- tence embeddings. Our embedding models are trained to produce similar representations ex- clusively for bilingual sentence pairs that are translations of each other. This is achieved using a novel training method that introduces hard negatives consisting of sentences that are not translations but have some degree of se- mantic similarity. The quality of the result- ing embeddings are evaluated on parallel cor- pus reconstruction and by assessing machine translation systems trained on gold vs. mined sentence pairs. We find that the sentence em- beddings can be used to reconstruct the United Nations Parallel Corpus (Ziemski et al., 2016) at the sentence-level with a precision of 48.9% for en-fr and 54.9% for en-es. When adapted to document-level matching, we achieve a par- allel document matching accuracy that is com- parable to the significantly more computa- tionally intensive approach of Uszkoreit et al. (2010). Using reconstructed parallel data, we are able to train NMT models that perform nearly as well as models trained on the orig- inal data (within 1-2 BLEU). 1 Introduction Volumes of quality parallel training data are criti- cal to neural machine translation (NMT) systems. While large distributed systems have proven use- ful for mining parallel documents (Uszkoreit et al., 2010; Antonova and Misyurev, 2011), these ap- proaches are computationally intensive and rely on heavily engineered subsystems. Recent work has approached the problem by training lightweight end-to-end models based on word and sentence- level embeddings (Gr´egoire and Langlais, 2017; Bouamor and Sajjad, 2018; Schwenk, 2018). We propose a novel method for training bilin- gual sentence embeddings that proves useful for ∗ equal contribution † Work done during an internship at Google AI. sentence-level mining of parallel data. Sen- tences are encoded using Deep Averaging Net- works (DANs) (Iyyer et al., 2015), a simple bag of n-grams architecture that has been shown to provide surprisingly competitive performance on a number of tasks including sentence classifica- tion (Iyyer et al., 2015; Cer et al., 2018), conversa- tion input-response prediction (Yang et al., 2018), and email response prediction (Henderson et al., 2017). Separate encoders are used for each lan- guage with candidate source and target sentences being paired based on the dot-product of their em- bedded representations. Training maximizes the dot-product score of sentence pairs that are trans- lations of each other at the expense of sampled negatives. We contrast using random negatives with carefully selected hard negatives that chal- lenge the model to distinguish between true trans- lation pairs versus non-translation pairs that ex- hibit …

Content truncated to 500 words