Pretrained Models for Multilingual Federated Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1413 - 1421 July 10-15, 2022 ©2022 Association for Computational Linguistics Pretrained Models for Multilingual Federated Learning Orion Weller*, Marc Marone*, Vladimir Braverman, Dawn Lawrie, Benjamin Van Durme Johns Hopkins University oweller@cs.jhu.edu,mmarone1@jhu.edu Abstract Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text im- pacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different lan- guages) on FL in naturally occurring data. We explore three multilingual language tasks, lan- guage modeling, machine translation, and text classification using differing federated and non- federated learning algorithms. Our results show that using pretrained models reduces the nega- tive effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.1 1 Introduction Federated learning (FL) is a machine learning tech- nique that trains a model across multiple distributed clients holding local data samples, without ever storing client data in a central location (Koneˇcn`y et al., 2016; McMahan et al., 2017). These tech- niques are appealing for those who wish to learn from data in a privacy-preserving way, without ever transmitting the data off of a client device. FL be- comes essential when data is especially sensitive, as is the case at hospitals, legal firms, financial institutions, or in countries that enact legislation concerning data privacy (such as the EU’s GDPR or the US’s HIPAA). FL has been applied to problems in natural lan- guage processing (NLP) since its inception, partic- ularly in use of the language modeling task (Yang et al., 2018; Hard et al., 2018; Ramaswamy et al., 2019; Chen et al., 2019a; Ji et al., 2019; Stremmel 1Our code and data are made publicly avail- able at https://github.com/orionw/ Multilingual-Federated-Learning * Authors contributed equally En Fr Ru Zh Model En Fr Ru Zh En Fr Ru Zh En Fr Ru Zh Zh Zh Zh Zh Model Ru Ru Ru Ru Fr Fr Fr Fr En En En En Client 1 Client 2 Client 3 Client 4 En x4 Fr x4 Ru x4 Zh x4 Federated Learning with IID Data Federated Learning with non-IID Data Centralized (standard) Learning Client 1 Client 2 Client 3 Client 4 Model Figure 1: A depiction of different learning strategies with Federated Learning (FL) and multilingual data, with 4 clients and 16 instances from En, Fr, Ru, and Zh in this toy example. Black lines indicate gradient flow. Centralized learning is the standard training method (no privacy), FL with IID data partitions the data into IID data subsets for each client, while FL with non-IID data has the languages separated across clients. and Singh, 2020). Another large area of FL re- search is focused on analyzing performance when the data is non identically independently distributed (non-IID). …

Content truncated to 500 words