Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings

Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings Phillip Keung• Julian Salazar• Yichao Lu• Noah A. Smith†‡ •Amazon †University of Washington ‡Allen Institute for AI {keung,julsal,yichaolu}@amazon.com nasmith@cs.washington.edu Abstract We describe an unsupervised method to create pseudo-parallel corpora for machine trans- lation (MT) from unaligned text. We use mul- tilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. We validate our technique by extracting par- allel sentence pairs on the BUCC 2017 bitext mining task and observe up to a 24.5 point increase (absolute) in F1 scores over previous unsupervised methods. We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation per- formance by up to 3.5 BLEU on the WMT’14 French-English and WMT’16 German-English tasks and outperforming the previous state- of-the-art. Finally, we enrich the IWSLT’15 English-Vietnamese corpus with pseudo- parallel Wikipedia sentence pairs, yielding a 1.2 BLEU improvement on the low-resource MT task. We demonstrate that unsupervised bitext mining is an effective way of augment- ing MT datasets and complements existing techniques like initializing with pre-trained contextual embeddings. 1 Introduction Large corpora of parallel sentences are prerequi- sites for training models across a diverse set of applications, such as neural machine translation (NMT; Bahdanau et al., 2015), paraphrase genera- tion (Bannard and Callison-Burch, 2005), and aligned multilingual sentence embeddings (Artetxe and Schwenk, 2019b). Systems that extract paral- lel corpora typically rely on various cross-lingual resources (e.g., bilingual lexicons, parallel cor- pora), but recent work has shown that unsuper- vised parallel sentence mining (Hangya et al., 2018) and unsupervised NMT (Artetxe et al., 2018; Lample et al., 2018a) produce surprisingly good results.1 Existing approaches to unsupervised parallel sentence (or bitext) mining start from bilingual word embeddings (BWEs) learned via an unsuper- vised, adversarial approach (Lample et al., 2018b). Hangya et al. (2018) created sentence represen- tations by mean-pooling BWEs over content words. To disambiguate semantically similar but non-parallel sentences, Hangya and Fraser (2019) additionally proposed parallel segment detection by searching for paired substrings with high simi- larity scores per word. However, using word embeddings to generate sentence embeddings ignores sentential context, which may degrade bitext retrieval performance. We describe a new unsupervised bitext mining approach based on contextual embeddings. We create sentence embeddings by mean-pooling the outputs of multilingual BERT (mBERT; Devlin et al., 2019), which is pre-trained on unaligned Wikipedia sentences across 104 languages. For a pair of source and target languages, we find candidate translations by using nearest-neighbor search with margin-based similarity scores bet- ween pairs of mBERT-embedded source and tar- get sentences. We bootstrap a dataset of positive and negative sentence pairs from these initial neighborhoods of candidates, then self-train mBERT on its own outputs. A final retrieval step gives a corpus of pseudo-parallel sentence pairs, which we expect to be a mix of actual translations and semantically related non-translations. 1By unsupervised, we mean that no cross-lingual resources like parallel text …

Content truncated to 500 words