NVIDIA NeMo’s Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 197–204 November 10–11, 2021. ©2021 Association for Computational Linguistics 197 NVIDIA NeMo’s Neural Machine Translation Systems for English ↔ German and English ↔ Russian News and Biomedical Tasks at WMT21 Sandeep Subramanian, Oleksii Hrinchuk, Virginia Adams, Oleksii Kuchaiev NVIDIA Santa Clara, CA {sandeepsub, ohrinchuk, vadams, okuchaiev}@nvidia.com Abstract This paper provides an overview of NVIDIA NeMo’s neural machine translation systems for the constrained data track of the WMT21 News and Biomedical Shared Translation Tasks. Our news task submissions for English ↔ German (En ↔ De) and English ↔ Rus- sian (En ↔ Ru) are built on top of a base- line transformer-based sequence-to-sequence model (Vaswani et al., 2017). Specifically, we use a combination of 1) checkpoint av- eraging 2) model scaling 3) data augmenta- tion with backtranslation and knowledge dis- tillation from right-to-left factorized models 4) finetuning on test sets from previous years 5) model ensembling 6) shallow fusion decoding with transformer language models and 7) noisy channel re-ranking. Additionally, our biomed- ical task submission for English ↔ Russian uses a biomedically biased vocabulary and is trained from scratch on news task data, medi- cally relevant text curated from the news task dataset, and biomedical data provided by the shared task. Our news system achieves a sacre- BLEU score of 39.5 on the WMT’20 En → De test set outperforming the best submission from last year’s task of 38.8. Our biomedical task Ru → En and En → Ru systems reach BLEU scores of 43.8 and 40.3 respectively on the WMT’20 Biomedical Task Test set, outper- forming the previous year’s best submissions. 1 Introduction We take part in the WMT’21 News Shared Task for English ↔ German, English ↔ Russian, and the Biomedical Shared Task for English ↔ Rus- sian. Our systems are implemented in the NVIDIA NeMo1 framework (Kuchaiev et al., 2019). They build on baseline sequence-to-sequence trans- former models (Vaswani et al., 2017) in the follow- ing ways: 1) Checkpoint averaging, 2) Model scal- ing up to 1B parameters, 3) Data augmentation with 1https://github.com/NVIDIA/NeMo large-scale backtranslation (Edunov et al., 2018) of monolingual Newscrawl data and sequence-level knowledge distillation from a right-to-left factor- ized model (Zhang et al., 2019b), 4) Finetuning models on in-domain news data from WMT test sets made available in previous years, 5) Ensem- bling models trained on different subsets of the overall data 6) Shallow fusion decoding with trans- former language models (Gulcehre et al., 2015) 7) Noisy channel re-ranking of beam search candidate hypotheses (Yee et al., 2019). Overall, we find each of these components re- sults in a small improvement in BLEU scores with backtranslation results being mixed depending on the language direction and whether the test data contains translationese inputs. Using a combina- tion of these techniques, we achieve 39.5 sacre- BLEU scores on the En → De WMT’20 test set, outperforming the best BLEU scores from last year’s competition of 38.77. Training our En ↔ Ru biomedical task submis- sion from scratch …

Content truncated to 500 words