Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12149–12163 July 9-14, 2023 ©2023 Association for Computational Linguistics Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms Tianshu Zhang1∗, Changchang Liu2, Wei-Han Lee2, Yu Su1, Huan Sun1 1The Ohio State University 2IBM Research 1{zhang.11535, su.809, sun.397}@osu.edu 2{changchang.liu33, wei-han.lee1}@ibm.com Abstract This paper studies a new task of federated learn- ing (FL) for semantic parsing, where multiple clients collaboratively train one global model without sharing their semantic parsing data. By leveraging data from multiple clients, the FL paradigm can be especially beneficial for clients that have little training data to develop a data-hungry neural semantic parser on their own. We propose an evaluation setup to study this task, where we re-purpose widely-used single-domain text-to-SQL datasets as clients to form a realistic heterogeneous FL setting and collaboratively train a global model. As stan- dard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel LOss Reduction Adjusted Re- weighting (Lorar) mechanism to mitigate the performance degradation, which adjusts each client’s contribution to the global model up- date based on its training loss reduction during each round. Our intuition is that the larger the loss reduction, the further away the current global model is from the client’s local optimum, and the larger weight the client should get. By applying Lorar to three widely adopted FL algorithms (FedAvg, FedOPT and FedProx), we observe that their performance can be im- proved substantially on average (4%-20% ab- solute gain under MacroAvg) and that clients with smaller datasets enjoy larger performance gains. In addition, the global model converges faster for almost all the clients.1 1 Introduction Semantic parsing aims to translate natural lan- guage utterances into formal meaning representa- tions such as SQL queries and API calls and can be applied to build natural language interfaces that en- able users to query data and invoke services without ∗Work started during the internship at IBM T. J. Watson Research Center and continued at OSU. 1Our code and data are publicly available at https://github.com/OSU-NLP-Group/ FL4SemanticParsing programming (Berant et al., 2013; Thomason et al., 2015; Su et al., 2017; Campagna et al., 2017). Neu- ral semantic parsers have achieved remarkable per- formance in recent years (Wang et al., 2020a; Rubin and Berant, 2021; Scholak et al., 2021). However, they are data-hungry; bootstrapping a neural se- mantic parser by annotating data on a large scale can be very challenging for many institutions, as it requires the annotators to have intimate knowl- edge of formal programs. One natural thought is to leverage data from different institutions and train a unified model that can be used for all institutions. However, in practice, institutions such as hospitals, banks, and legal firms are prohibited from shar- ing their data with others, due to privacy concerns. Therefore, for institutions that only have very lim- ited data, it is extremely hard to build their own neural semantic parsers. Federated learning (FL) …

Content truncated to 500 words