FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2118–2128, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2118 FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction Dianbo Sui♥ ♠ , Yubo Chen♥ ♠, Jun Zhao♥ ♠, Yantao Jia♦, Yuantao Xie♦, Weijian Sun♦ ♥ National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China ♠ School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China ♦ Huawei Technologies Co., Ltd., Beijing, China {dianbo.sui, yubo.chen, jzhao}@nlpr.ia.ac.cn, jamaths.h@163.com {xieyuantao2, sunweijian}@huawei.com Abstract Unlike other domains, medical texts are in- evitably accompanied by private information, so sharing or copying these texts is strictly restricted. However, training a medical rela- tion extraction model requires collecting these privacy-sensitive texts and storing them on one machine, which comes in conflict with privacy protection. In this paper, we propose a privacy- preserving medical relation extraction model based on federated learning, which enables training a central model with no single piece of private local data being shared or exchanged. Though federated learning has distinct advan- tages in privacy protection, it suffers from the communication bottleneck, which is mainly caused by the need to upload cumbersome lo- cal parameters. To overcome this bottleneck, we leverage a strategy based on knowledge dis- tillation. Such a strategy uses the uploaded pre- dictions of ensemble local models to train the central model without requiring uploading lo- cal parameters. Experiments on three publicly available medical relation extraction datasets demonstrate the effectiveness of our method. 1 Introduction Privacy - like eating and breathing - is one of life’s basic requirements. — Katherine Neville Relation extraction is a task of mining factual knowledge from the free text by labeling relations between entity mentions and has attracted increas- ing attention in recent years, such as Zeng et al. (2014); Xu et al. (2015a,b); Wang et al. (2016); Baldini Soares et al. (2019); Song et al. (2019). Applying automatic relation extraction to medical texts, such as electronic health records and dis- charge summaries, can be useful for many appli- cations, including drug repurposing and medical knowledge graph construction. Unlike other domains, medical texts are highly privacy-sensitive, because these texts can include some of the most intimate details about one’s life, which document a patient’s physical and mental health, and can include information on social be- haviors, personal relationships and financial status (Gostin and Hodge, 2002). To prevent private infor- mation leakage, sharing or copying medical texts is strictly restricted. Previous relation extraction methods require cen- tralizing the underlying training data from different medical platforms, such as hospitals and healthcare centers, on one server for training, while holding the centralized privacy-sensitive data puts patients’ privacy at risk. This is one of the reasons that hin- der the use of relation extraction in clinical practice. As a possible solution, federated learning (McMa- han et al., 2016) is proposed to make full use of privacy-sensitive data. Training local models with private data at local platforms and aggregating local models …

Content truncated to 500 words