Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages

Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages Dana Ruiter druiter@lsv.uni-saarland.de Dietrich Klakow dietrich.klakow@lsv.uni-saarland.de Spoken Language Systems Group, Saarland University, Germany Josef van Genabith josef.van genabith@dfki.de DFKI GmbH & Saarland University, Saarland Informatics Campus, Saarbr¨ucken, Germany Cristina Espa˜na-Bonet cristinae@dfki.de DFKI GmbH, Saarland Informatics Campus, Saarbr¨ucken, Germany Abstract For most language combinations, parallel data is either scarce or simply unavailable. To ad- dress this, unsupervised machine translation (UMT) exploits large amounts of monolingual data by using synthetic data generation techniques such as back-translation and noising, while self-supervised NMT (SSNMT) identifies parallel sentences in smaller comparable data and trains on them. To date, the inclusion of UMT data generation techniques in SSNMT has not been investigated. We show that including UMT techniques into SSNMT significantly out- performs SSNMT and UMT on all tested language pairs, with improvements of up to +4.3 BLEU, +50.8 BLEU, +51.5 over SSNMT, statistical UMT and hybrid UMT, respectively, on Afrikaans to English. We further show that the combination of multilingual denoising au- toencoding, SSNMT with backtranslation and bilingual finetuning enables us to learn machine translation even for distant language pairs for which only small amounts of monolingual data are available, e.g. yielding BLEU scores of 11.6 (English to Swahili). 1 Introduction Neural machine translation (NMT) achieves high quality translations when large amounts of parallel data are available (Barrault et al., 2020). Unfortunately, for most language combina- tions, parallel data is non-existent, scarce or low-quality. To overcome this, unsupervised MT (UMT) (Lample et al., 2018b; Ren et al., 2019; Artetxe et al., 2019) focuses on exploiting large amounts of monolingual data, which are used to generate synthetic bitext training data via var- ious techniques such as back-translation or denoising. Self-supervised NMT (SSNMT) (Ruiter et al., 2019) learns from smaller amounts of comparable data –i.e. topic-aligned data such as Wikipedia articles– by learning to discover and exploit similar sentence pairs. However, both UMT and SSNMT approaches often do not scale to low-resource languages, for which nei- ther monolingual nor comparable data are available in sufficient quantity (Guzm´an et al., 2019; Espa˜na-Bonet et al., 2019; Marchisio et al., 2020). To date, UMT data augmentation techniques have not been explored in SSNMT. However, both approaches can benefit from each other, as i) SSNMT has strong internal quality checks on the data it admits for training, which can be Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 76 of use to filter low-quality synthetic data, and ii) UMT data augmentation makes monolingual data available for SSNMT. In this paper we explore and test the effect of combining UMT data augmentation with SSNMT on different data sizes, ranging from very low-resource (∼ 66k non-parallel sentences) to high-resource (∼ 20M sentences). We do this using a common high-resource language pair (en–fr), which we downsample while keeping all other parameters identical. We then proceed to evaluate the augmentation techniques on different truly low-resource similar and distant lan- guage …

Content truncated to 500 words