FEDLEGAL: The First Real-World Federated Learning Benchmark for Legal NLP

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3492–3507 July 9-14, 2023 ©2023 Association for Computational Linguistics FEDLEGAL: The First Real-World Federated Learning Benchmark for Legal NLP Zhuo Zhang1,2,∗ Xiangjing Hu1,∗ Jingyuan Zhang4 Yating Zhang4 Hui Wang2 Lizhen Qu3,† Zenglin Xu1,2,† 1Harbin Institute of Technology, Shenzhen, China 2Peng Cheng Lab, Shenzhen, China 3Monash University, Melbourne, Australia 4Independent Researcher {iezhuo17, starry.hxj, zhangjingyuan1994, yatingz89}@gmail.com wanghu06@pcl.ac.cn Lizhen.Qu@monash.edu.cn xuzenglin@hit.edu.cn Abstract The inevitable private information in legal data necessitates legal artificial intelligence to study privacy-preserving and decentralized learning methods. Federated learning (FL) has merged as a promising technique for multiple partici- pants to collaboratively train a shared model while efficiently protecting the sensitive data of participants. However, to the best of our knowl- edge, there is no work on applying FL to legal NLP. To fill this gap, this paper presents the first real-world FL benchmark for legal NLP, coined FEDLEGAL, which comprises five legal NLP tasks and one privacy task based on the data from Chinese courts. Based on the exten- sive experiments on these datasets, our results show that FL faces new challenges in terms of real-world non-IID data. The benchmark also encourages researchers to investigate privacy protection using real-world data in the FL set- ting, as well as deploying models in resource- constrained scenarios. The code and datasets of FEDLEGAL are available here. 1 Introduction It has been noticed that learning, comprehending and properly using an ever-increasing huge amount of legal data is way beyond human capability of legal practitioners (Gomes et al., 2022). Since the majority of the data is text, such an “information crisis in law” is encouraging the research and de- velopment of legal Natural Language Processing (NLP) techniques, to provide affordable legal ser- vices to both legal professionals and the general public (Sun et al., 2020a). As the majority of those techniques are based on machine learning, they re- quire training on centralized datasets. However, such approaches raise increasing privacy concerns *Equal contribution. †Corresponding authors. Pre-trained Language Models Participant 1 Participant 2 Participant N Server ... =∑(#####, #### , … , ### ) Legal Judgment Prediction Legal Argument Mining Legal Cause Prediction Legal Entity Recognition Local Sensitive Data Updated Model Parameters Local Legal Task Legal Relation Extraction Legal Privacy Attack Figure 1: The overview of FEDLEGAL. of the public and impose risks of breaching data protection laws, such as the General Data Protec- tion Regulation (GDPR). To address the above concerns, federated learn- ing (FL) is widely considered as a family of train- ing algorithms to achieve a promising trade-off between information utility and privacy preserva- tion, without sharing sensitive data of data own- ers (McMahan et al., 2017). As depicted in Fig- ure 1, those algorithms permit local machines of participants to coordinate with one or multiple servers to train a model in a decentralized and col- laborative way while preserving data privacy. De- spite its rosy future, FL still faces open challenges due to the needs of …

Content truncated to 500 words