Empirical Studies of Institutional Federated Learning For Natural Language Processing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 625–634 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 625 Empirical Studies of Institutional Federated Learning For Natural Language Processing Xinghua Zhu, Jianzong Wang � †, Zhenhou Hong, and Jing Xiao Ping An Technology (Shenzhen) Co., Ltd. Shenzhen, P.R.China {zhuxinghua889, wangjianzong347, hongzhenhou168, xiaojing661}@pingan.com.cn Abstract Federated learning has sparkled new interests in the deep learning society to make use of isolated data sources from independent insti- tutes. With the development of novel train- ing tools, we have successfully deployed fed- erated natural language processing networks on GPU-enabled server clusters. This paper demonstrates federated training of a popular NLP model, TextCNN, with applications in sentence intent classification. Furthermore, differential privacy is introduced to protect par- ticipants in the training process, in a man- ageable manner. Distinguished from previous client-level privacy protection schemes, the proposed differentially private federated learn- ing procedure is defined in the dataset sample level, inherent with the applications among in- stitutions instead of individual users. Optimal settings of hyper-parameters for the federated TextCNN model are studied through compre- hensive experiments. We also evaluated the performance of federated TextCNN model un- der imbalanced data load configuration. Experiments show that, the sampling ratio has a large impact on the performance of the FL models, causing up to 38.4% decrease in the test accuracy, while they are robust to differ- ent noise multiplier levels, with less than 3% variance in the test accuracy. It is also found that the FL models are sensitive to data load balancedness among client datasets. When the data load is imbalanced, model performance dropped by up to 10%. 1 Introduction Federated learning is a promising ideology to unite isolated datasets for machine learning problems (Koneˇcn`y et al., 2016; McMahan et al., 2016; Zhu et al., 2019). In the federated learning framework, no raw data are exchanged among participating en- tities. Instead, parameter gradients and aggregated † Cooresponding author: Jianzong Wang (jzwang@188.com) updates are communicated between servers during collective optimization. Therefore, without leaking private information, institutes can cooperate with each other by contributing their data collection in the training of a unified model. Such a feature is es- pecially desirable when handling sensitive data that involve e.g. personal preference, financial transac- tions, medical records, etc. An example of success- ful deployment of federated learning is the smart input prediction in Google Input (Hard et al., 2018). In addition, more business-to-client model training applications are drawing intensive attention of the public (Lim et al., 2020; Yang et al., 2020; Kong et al., 2020). Apart from this business-to-client cooperation case, more interesting applications can be found among institutions. Potential areas in- clude medical image analysis (Sheller et al., 2018), smart retail (Yang et al., 2019b), fraud detection, etc. Despite its promising designs, federated learn- ing met quite some difficulties migrating to deeper neural networks, as well as to broader cooperative areas. These difficulties are largely due to 1. the limited training speed offered by a secured federated learning …

Content truncated to 500 words