Improving Parallel Data Identification using Iteratively Refined Sentence Alignments and Bilingual Mappings of Pre-trained Language Models

Proceedings of the 5th Conference on Machine Translation (WMT), pages 972–978 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 972 Improving Parallel Data Identification using Iteratively Refined Sentence Alignments and Bilingual Mappings of Pre-trained Language Models Chi-kiu Lo and Eric Joanis Multilingual Text Processing Digital Technologies Research Centre National Research Council Canada (NRC-CNRC) 1200 Montreal Road, Ottawa, ON K1A 0R6, Canada {chikiu.lo,eric.joanis}@nrc-cnrc.gc.ca Abstract The National Research Council of Canada’s team submissions to the parallel corpus fil- tering task at the Fifth Conference on Ma- chine Translation are based on two key com- ponents: (1) iteratively refined statistical sen- tence alignments for extracting sentence pairs from document pairs and (2) a crosslingual semantic textual similarity metric based on a pretrained multilingual language model, XLM- RoBERTa, with bilingual mappings learnt from a minimal amount of clean parallel data for scoring the parallelism of the extracted sen- tence pairs. The translation quality of the neu- ral machine translation systems trained and fine-tuned on the parallel data extracted by our submissions improved significantly when compared to the organizers’ LASER-based baseline, a sentence-embedding method that worked well last year. For re-aligning the sen- tences in the document pairs (component 1), our statistical approach has outperformed the current state-of-the-art neural approach in this low-resource context. 1 Introduction The aim of the Fifth Conference on Machine Trans- lation (WMT20) shared task on parallel corpus filtering (Koehn et al., 2020) is essentially the same as the two previous editions (Koehn et al., 2018b, 2019): identifying high-quality sentence pairs in a noisy corpus crawled from the web using ParaCrawl (Koehn et al., 2018a), in order to train machine translation (MT) systems on the clean data. This year, the low-resource language pairs being tested are Khmer–English (km–en) and Pashto– English (ps–en). Specifically, participating sys- tems must produce a score for each sentence pair in the test corpora indicating the quality of that pair. Then samples containing the top-scoring 5M words are used to train MT systems. While using the filtered parallel data to train a FAIRseq (Ott et al., 2019) neural machine translation (NMT) sys- tem remains the same as last year, the organisers are no longer building statistical machine transla- tion (SMT) systems as part of the task evaluation. Instead, as an alternative evaluation, the filtered parallel corpus is used to fine-tune an MBART (Liu et al., 2020) pretrained NMT system. Partic- ipants were ranked based on the performance of these MT systems on a test set of Wikipedia transla- tions (Guzmán et al., 2019), as measured by BLEU (Papineni et al., 2002). A few small sources of parallel data, covering different domains, were pro- vided for each of the two low-resource languages. Much larger monolingual corpora were also pro- vided for each language (en, km and ps). In ad- dition to the task of computing quality scores for the purpose of filtering, there is also a sub-task of re-aligning the sentence pairs from the original crawled document pairs. Cleanliness or quality of parallel corpora for MT systems …

Content truncated to 500 words