A Secure and Efficient Federated Learning Framework for NLP

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7676–7682 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7676 A Secure and Efficient Federated Learning Framework for NLP Jieren Deng1§, Chenghong Wang2§, Xianrui Meng3, Yijue Wang1, Ji Li4, Sheng Lin5 Shuo Han6, Fei Miao1, Sanguthevar Rajasekaran1, Caiwen Ding1 1University of Connecticut, 2Duke University, 3Facebook, 4Microsoft 5Northeastern University, 6University of Illinois at Chicago {jieren.deng,yijue.wang,fei.miao,sanguthevar.rajasekaran,caiwen.ding}@uconn.edu chenghong.wang552@duke.edu,{xianruimeng,changzhouliji}@gmail.com lin.sheng@northeastern.edu, hanshuo@uic.edu Abstract In this work, we consider the problem of de- signing secure and efficient federated learn- ing (FL) frameworks. Existing solutions ei- ther involve a trusted aggregator or require heavyweight cryptographic primitives, which degrades performance significantly. More- over, many existing secure FL designs work only under the restrictive assumption that none of the clients can be dropped out from the training protocol. To tackle these problems, we propose SEFL, a secure and efficient FL framework that (1) eliminates the need for the trusted entities; (2) achieves similar and even better model accuracy compared with existing FL designs; (3) is resilient to client dropouts. Through extensive experimental studies on natural language processing (NLP) tasks, we demonstrate that the SEFL achieves compa- rable accuracy compared to existing FL solu- tions, and the proposed pruning technique can improve runtime performance up to 13.7×. 1 Introduction Deep Neural Networks have played a significant role in advancing many applications (Yuan et al., 2021; Ding et al., 2017). The field of Natural Lan- guage Processing (NLP) leverages Recurrent Neu- ral Networks (RNNs) and Transformers to achieve outstanding performance on many tasks. The Trans- former was first introduced in (Vaswani et al., 2017) using a self-attention mechanism and it achieved prominent performance in various NLP tasks. The benefits of RNNs and Transformers in NLP are well-publicized, but the various privacy and secu- rity problems still pose challenges to the utilization of these models by data owners, especially users with sensitive data such as location, health, and fi- nancial datasets. Federated Learning (FL) (McMa- han et al., 2017a) empowers different data owners §Equal contribution, alphabetical order (e.g., organizations or edge devices) to collabora- tively train a model without sharing their own data, thus allowing them to address key issues like data privacy. Although data exchanged in FL consists of less information of the user’s raw data (Bonawitz et al., 2019), one might still be concerned about how much information remains. Recent research has shown that attackers can still infer sensitive information about the training data, or even recon- struct the it solely from publicly shared model pa- rameters. (Zhu et al., 2019). Although a series of works (Bonawitz et al., 2017; Truex et al., 2019; Papernot et al., 2018; Wu et al., 2021; Lin et al., 2020) have been proposed to protect FL protocols from leaking sensitive infor- mation (Wang et al., 2021; Deng et al., 2021; Wang et al., 2020). They either have to involve a trusted third party (centralized aggregator), or do not tol- erate client dropouts. Therefore, the data owners either need to blindly trust …

Content truncated to 500 words