Training Mixed-Domain Translation Models via Federated Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2576 - 2586 July 10-15, 2022 ©2022 Association for Computational Linguistics Training Mixed-Domain Translation Models via Federated Learning Peyman Passban∗ Tanya Roosta∗ Rahul Gupta Ankit Chadha Clement Chung Amazon {peymp,troosta,gupra,ankitrc,chungcle}@amazon.com Abstract Training mixed-domain translation models is a complex task that demands tailored architec- tures and costly data preparation techniques. In this work, we leverage federated learning (FL) in order to tackle the problem. Our investiga- tion demonstrates that with slight modifications in the training process, neural machine trans- lation (NMT) engines can be easily adapted when an FL-based aggregation is applied to fuse different domains. Experimental results also show that engines built via FL are able to perform on par with state-of-the-art baselines that rely on centralized training techniques. We evaluate our hypothesis in the presence of five datasets with different sizes, from different domains, to translate from German into English and discuss how FL and NMT can mutually benefit from each other. In addition to provid- ing benchmarking results on the union of FL and NMT, we also propose a novel technique to dynamically control the communication band- width by selecting impactful parameters during FL updates. This is a significant achievement considering the large size of NMT engines that need to be exchanged between FL parties. 1 Introduction Federated learning (FL) is a rapidly growing field in the machine learning community. The reason for this popularity is because of its decentralized and private nature. Model training in FL is distributed over multiple nodes where each node could be an independent piece of hardware with its own iso- lated data. This unique feature enables building high-quality models that benefit from external re- sources without requiring access to them. Although FL is a relatively new field (McMa- han et al., 2017), it has drawn researchers’ atten- tion and the community has witnessed a rapid growth. Fields such as computer vision have adapted quickly to the FL framework (Geyer et al., ∗Equal contribution. 2017; Lin et al., 2018; Hardy et al., 2019; Geiping et al., 2020; Ren et al., 2021), but others, such as natural language processing (NLP), have not been as quick and only recently have begun to explore (Wu et al., 2020). We believe that the reason for this slow(er) in- tegration of NLP and FL is because representa- tion learning in NLP is complicated and down- stream tasks require large and data-hungry models. These requirements are heavy-handed for any FL model and slow down the unification. However, real-world NLP problems necessitate the use of distributed solutions with privacy-preserving char- acteristics (Li et al., 2020). Our work is an effort towards leveraging FL-based solutions in NLP. In this paper, we focus on NMT to combine it with FL. A review of the NLP literature shows that almost all recent groundbreaking architectures have been first proposed, or at least evaluated, for translation (Bahdanau et al., 2014; Sutskever et al., 2014; Gehring …

Content truncated to 500 words