Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3685 - 3695 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation Kevin Yang1,∗ Olivia Deng2 Charles Chen2 Richard Shin2 Subhro Roy2 Benjamin Van Durme2 1UC Berkeley, 2Microsoft Semantic Machines yangk@berkeley.edu, sminfo@microsoft.com Abstract We introduce a novel setup for low-resource task-oriented semantic parsing which incor- porates several constraints that may arise in real-world scenarios: (1) lack of similar datasets/models from a related domain, (2) in- ability to sample useful logical forms directly from a grammar, and (3) privacy requirements for unlabeled natural utterances. Our goal is to improve a low-resource semantic parser us- ing utterances collected through user interac- tions. In this highly challenging but realistic setting, we investigate data augmentation ap- proaches involving generating a set of struc- tured canonical utterances corresponding to logical forms, before simulating correspond- ing natural language and filtering the result- ing pairs. We find that such approaches are effective despite our restrictive setup: in a low- resource setting on the complex SMCalFlow calendaring dataset (Andreas et al., 2020), we observe 33% relative improvement over a non- data-augmented baseline in top-1 match. 1 Introduction We aim to improve the performance of a seman- tic parser based on previous user interactions, but without making use of their direct utterances, nor any associated personal identifiable information (PII). Such privacy requirements are common in practical deployment (Kannan et al., 2016), and semantic parsers are commonly used in real-world systems such as Siri and Alexa, converting natu- ral language into structured queries to be executed downstream (Kamath and Das, 2018). Constructing semantic parsers can be expensive: annotating examples consisting of natural language- logical form pairs often requires trained experts. Two complementary lines of work has pursued this concern. First, several works (Zhong et al., 2020; Cao et al., 2020) tackle low-resource semantic pars- ing via approaches such as data augmentation. A 1*: Work done during internship at Semantic Machines. Natural When is Allison’s birthday? Logical (Yield :output (:start (singleton (:results (FindEventWrapperWithDefaults :con- straint (Constraint[Event] :subject (? = #(String “Allison’s birthday”)))))))) Canonical start time of find event called something like “Allison’s birthday” Table 1: An example of natural language, logical form, and canonical form in the SMCalFlow domain. The event title, “Allison’s birthday,” is private information. second line of work (Wang et al., 2015; Xiao et al., 2016) explores canonical utterances: structured language which maps directly to logical forms, but resembles natural language (Table 1). The use of canonical forms as the target of semantic parsing has shown benefits in accuracy (Shin et al., 2021; Wu et al., 2021). We consider low-resource semantic parsing with further resource and privacy constraints which may arise in practical deployment: beyond a small gold dataset of labeled pairs, we assume only unlabeled natural utterances which must be masked for PII. Unlike many prior works, we assume that (1) we do not have a large dataset of related logical forms in a different domain, (2) …

Content truncated to 500 words