Practical Takes on Federated Learning with Pretrained Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 454–471 May 2-6, 2023 ©2023 Association for Computational Linguistics Practical Takes on Federated Learning with Pretrained Language Models Ankur Agarwal Mehdi Rezagholizadeh Prasanna Parthasarathi Huawei Noah’s Ark Lab, Montréal {ankur.agarwal1,mehdi.rezagholizadeh,prasanna.parthasarathi}@huawei.com Abstract Real-world applications of language models entail data privacy constraints when learning from diverse data domains. Federated learn- ing with pretrained language models for lan- guage tasks has been gaining attention lately but there are definite confounders that warrants a careful study. Specifically, understanding the limits of federated NLP applications through varying the effects of different aspects (such as data heterogeneity, the trade-off between training time and performance, the effect of different data, and client distributions and sen- sitivity of the shared model to learning local distributions) is necessary to evaluate whether language models indeed learn to generalize by adapting to the different domains. Towards that, we elaborate different hypotheses over the components in federated NLP architectures and study them in detail with relevant experiments over three tasks: Stanford Sentiment Treebank- 2, OntoNotes-5.0 and GigaWord. The experi- ments with different Transformer inductive bi- ases on the variety of tasks provide a glimpse at the understanding of federated learning at NLP tasks. Specifically, the analysis suggests that regularization due to the ensembling effect may be masquerading as domain adaptation of feder- ated learning in NLP with pre-trained language models. 1 Introduction The success of large pretrained language models (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Lewis et al., 2019) have demonstrated their applicability in consumer-based natural language processing (NLP) applications (Otter et al., 2020). While there are such massive datasets(Kiela et al., 2021; Wang et al., 2021), making models trained on these datasets to reflect the data diversity is an im- portant challenge towards building equitable NLP systems. Hence, treating the distribution of data over the users as non-IID (McMahan and Ramage, 2017; Xu et al., 2018; Liu and Mazumder, 2021) to better emphasize the preferences of users as person- alization gets naturally extended to the consumer NLP applications. But, recent studies highlight that pretrained lan- guage models (PLMs) (Devlin et al., 2019) tend to get their predictions skewed by the frequency effects of tokens in the data distribution (Wei et al., 2021), this is concerning from a privacy and person- alization standpoint. Han and Eisenstein (2019); Ramponi and Plank (2020); Carlini et al. (2022) show that neural language models (even the large pretrained architectures) have challenges in adapt- ing to different data distributions on generative and classification tasks alike. Federated learning (FL) (McMahan and Ram- age, 2017; Koneˇcn`y et al., 2016) has been gaining popularity in machine learning as a practical way to mitigate domain adaptation with the promise of data privacy. FL as a learning paradigm focuses on learning a shared model through training data distributed over several clients. Such approaches have only recently begun to focus on NLP applica- tions (Mammen, 2021; Lin et al., 2021). Lin et al. (2021) suggest that the …

Content truncated to 500 words