Federated Model Decomposition with Private Vocabulary for Text Classification

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6413 - 6425 December 7-11, 2022 ©2022 Association for Computational Linguistics Federated Model Decomposition with Private Vocabulary for Text Classification Zhuo Zhang♡♠ Xiangjing Hu♡ Lizhen Qu♣∗ Qifan Wang♢ Zenglin Xu♡♠ ♡Harbin Institute of Technology (Shenzhen), China ♠Peng Cheng Lab, Shenzhen, China ♣Monash University, Melbourne, Australia ♢Meta AI, CA, USA {iezhuo17,starry.hxj}@gmail.com Lizhen.Qu@monash.edu wqfcr@fb.com xuzenglin@hit.edu.cn Abstract With the necessity of privacy protection, it be- comes increasingly vital to train deep neural models in a federated learning manner for nat- ural language processing (NLP) tasks. How- ever, recent studies show eavesdroppers (i.e., dishonest servers) can still reconstruct the pri- vate input in federated learning (FL). Such a data reconstruction attack relies on the map- pings between vocabulary and associated word embedding in NLP tasks, which are unfor- tunately less studied in current FL methods. In this paper, we propose a fedrated model decomposition method that protects the privacy of vocabularies, shorted as FEDEVOCAB. In FEDEVOCAB, each participant keeps the local embedding layer in the local device and de- taches the local embedding parameters from federated aggregation. However, it is chal- lenging to train an accurate NLP model when the private mappings are unknown and vary across participants in a cross-device FL set- ting. To address this problem, we further pro- pose an adaptive updating technique to improve the performance of local models. Experimen- tal results show that FEDEVOCAB maintains competitive performance and provides better privacy-preserving capacity compared to status quo methods. 1 Introduction Privacy-sensitive Natural Language Processing (NLP) applications, such as personal virtual as- sistants (Chen et al., 2017), online medical diagno- sis (Hakak et al., 2020) and mobile keyboards (Ji et al., 2019), often have sensitive user data stored in local devices to protect user privacy. To ensure service quality, they also need a large amount of such data for training the corresponding machine learning models. However, it would breach user privacy or data protection law, e.g., GDPR, to apply conventional training methods by putting sensitive data in a centralized place. ∗Co-corresponding author Hi, recently I feel very thirsty and urinate more. And I get hungry easily and eat more than usual. But being 165cm tall, my weight even dropped to 40kg. It has continued for half a month. Hi! Does your family have a history of diabetes? If so, there is a high probability of diabetes. Personal health status User Privacy Data Reconstruction …, thirsty, 165cm, …, live, ingram, york, 123456, phone, …, district, urinate, hungry, …, blood, sugar, street, 212- 17564321, …, 40kg, forest, number, …, diabetes, history, probability, grandma, … Address Let me see…Uhh, my grandma has diabetes. This bonus is great! And I want one. I live at 123456 Ingram Street, Forest District, New York. And my phone number is 212-17654321. Thanks! Our platform is promoting products such as blood glucose meters. If you need to monitor your blood sugar, you can get this bonus for free by leaving your delivery address. Family disease history …

Content truncated to 500 words