Facebook AI’s WMT21 News Translation Task Submission

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 205–215 November 10–11, 2021. ©2021 Association for Computational Linguistics 205 Facebook AI’s WMT21 News Translation Task Submission Chau Tran Shruti Bhosale James Cross Philipp Koehn Sergey Edunov Angela Fan Facebook AI chau,shru,jcross,pkoehn,edunov,angelafan@fb.com Abstract We describe Facebook’s multilingual model submission to the WMT2021 shared task on news translation. We participate in 14 lan- guage directions: English to and from Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese. To develop systems covering all these directions, we focus on multilingual models. We utilize data from all available sources — WMT, large-scale data mining, and in-domain backtranslation — to create high quality bilingual and multilingual baselines. Subsequently, we investigate strategies for scaling multilingual model size, such that one system has sufficient capacity for high quality representations of all eight languages. Our fi- nal submission is an ensemble of dense and sparse Mixture-of-Expert multilingual transla- tion models, followed by finetuning on in- domain news data and noisy channel reranking. Compared to previous year’s winning submis- sions, our multilingual system improved the translation quality on all language directions, with an average improvement of 2.0 BLEU. In the WMT2021 task, our system ranks first in 10 directions based on automatic evaluation. 1 Introduction We participate in the WMT2021 shared task on news translation and submit a multilingual transla- tion system. In recent years, multilingual transla- tion has gained significant interest as an alternative to developing separate, specialized systems for dif- ferent language directions (Firat et al., 2016; Tan et al., 2019; Aharoni et al., 2019; Zhang et al., 2020; Tang et al., 2020; Arivazhagan et al., 2019). Multilingual systems have great potential for sim- plicity and consolidation, making them attractive options for the development and maintenance of commercial translation technologies. From a re- search standpoint, studies of transfer learning be- tween related languages and developing methods that incorporate low-resource languages are strong motivators for grouping languages together in one system (Dabre et al., 2019; Fan et al., 2021). Despite such motivations, existing multilingual translation systems have been unable to show that the translation quality of multilingual systems sur- passes that of bilingual. Several works compare to bilingual baselines, but these baselines do not in- corporate standard techniques used across the field — such as backtranslation or dense model scaling. Further, multilingual translation systems are often developed on non-standard training datasets and use different evaluation datasets. These factors make it difficult to assess the performance of multi- lingual translation, particularly when compared to the most competitive bilingual models. In this work, our aim is to demonstrate against the winning WMT2020 models and our bilingual WMT2021 systems that multilingual translation models have stronger performance than bilingual ones. We focus on 14 language directions: En- glish to and from Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese. We create an un- constrained system that utilizes both WMT dis- tributed and publicly available training data, apply large-scale backtranslation, and explore dense and mixture-of-expert architectures. We compare the impact of various techniques …

Content truncated to 500 words