Privacy-Preserving Domain Adaptation of Semantic Parsers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4950–4970 July 9-14, 2023 ©2023 Association for Computational Linguistics Privacy-Preserving Domain Adaptation of Semantic Parsers Fatemehsadat Mireshghallah1,2∗ Yu Su2 Tatsunori Hashimoto2 Jason Eisner2 Richard Shin2 1 University of California, San Diego 2 Microsoft Semantic Machines fatemeh@ucsd.edu {yusu2,v-hashimotot,jason.eisner,richard.shin}@microsoft.com Abstract Task-oriented dialogue systems often assist users with personal or confidential matters. For this reason, the developers of such a system are generally prohibited from observing actual usage. So how can they know where the system is failing and needs more training data or new functionality? In this work, we study ways in which realistic user utterances can be generated synthetically, to help increase the linguistic and functional coverage of the system, without compromising the privacy of actual users. To this end, we propose a two-stage Differentially Private (DP) generation method which first generates latent semantic parses, and then generates utterances based on the parses. Our proposed approach improves MAUVE by 2.5× and parse tree function type overlap by 1.3× rel- ative to current approaches for private synthetic data generation, improving both on fluency and semantic coverage. We further validate our ap- proach on a realistic domain adaptation task of adding new functionality from private user data to a semantic parser, and show overall gains of 8.5% points in accuracy with the new feature. 1 Introduction In task-oriented dialogue systems, such as Siri and Alexa, a software agent parses a user’s intent into a program, executes it and then communicates the results back to the user (Andreas et al., 2020; Li et al., 2022; Cheng et al., 2020; Gupta et al., 2018; Young et al., 2013). As a result of their growing popularity, these systems face an increasing demand to improve their linguistic coverage (How do users talk?) as well as functional coverage (What are users trying to do?). An input utterance to such a system could look like this: “Could you tell me what the weather is gonna be like today in ∗ Work done as part of a Microsoft Semantic Machines internship. Corresponding author email: fatemeh@ucsd.edu New York?” and the agent answers it by predicting and executing a program, or semantic parse. In many cases, due to privacy controls, system developers can only use datasets that are limited and contrived, e.g., dialogues created by crowd workers pretending to be users. This is a significant domain shift from real private user data. Unlabeled data from real user interactions with dialogue systems has abundant signals that could be used to improve the linguistic and functional coverage of semantic parsers. For instance, practitioners could detect gaps in coverage by examining interactions where the system was unsure about what to do or responded that it lacked the requested functionality. Note that training on real user interactions would be problematic for privacy even if automated and unsupervised. Trained models can “memorize” details of their training data (Tirumala et al., 2022; Feldman, 2020; Mireshghallah et al., 2022b), and this can …

Content truncated to 500 words