Tailor: Generating and Perturbing Text with Semantic Controls

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3194 - 3213 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Tailor: Generating and Perturbing Text with Semantic Controls Alexis Ross∗∗† Tongshuang Wu∗♦ Hao Peng♦ Matthew E. Peters† Matt Gardner♠† †Allen Institute for Artificial Intelligence, Seattle, WA, USA ♦Paul G. Allen School of Computer Science and Engineering, University of Washington ♠Microsoft Semantic Machines, USA {alexisr,matthewp}@allenai.org {wtshuang,hapeng}@cs.washington.edu mattgardner@microsoft.com Abstract Controlled text perturbation is useful for eval- uating and improving model generalizability. However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize. We present Tailor, a semantically-controlled text gener- ation system. Tailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from se- mantic representations. We craft a set of op- erations to modify the control codes, which in turn steer generation towards targeted at- tributes. These operations can be further com- posed into higher-level ones, allowing for flex- ible perturbation strategies. We demonstrate the effectiveness of these perturbations in mul- tiple applications. First, we use Tailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks. These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical di- versity. Second, we show that Tailor per- turbations can improve model generalization through data augmentation. Perturbing just ∼2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics. 1 Introduction Semantic perturbation through controlled text gen- eration modifies sentences to match certain target attributes, such as verb tense or sentiment (e.g., pos- itive→negative). It has been widely applied to a variety of tasks, e.g., changing text style (Reid and Zhong, 2021), mitigating dataset biases (Gardner et al., 2021), explaining model behaviors (Ross et al., 2021), and improving model generaliza- tion (Teney et al., 2020; Wu et al., 2021). Existing efforts train task-specific generators, e.g., training ∗ denotes equal contribution. † Work done while at Allen Institute for AI LOCATIVE→TEMPORAL+partial: in LOCATIVE In the operation room AGENT the doctor VERB comforted , PATIENT the athlete . [ | | ] <id_0>, the doctor <id_2> <id_3>. [TEMPORAL: In the midst of the earthquake], the doctor [VERB: is comforting][PATIENT: the athlete panicking]. PATIENT+complete→partial: the athlete VERB+active+past→present: comfort A D B Input Output C LOCATIVE:CHANGE_TAG(TEMPORAL) VERB:CHANGE_VTENSE(present) PATIENT:CHANGE_SPEC(sparse) Figure 1: A compositional perturbation using Tai- lor.1 Given (A) an original sentence, we abstract each span into a structured header that contains its seman- tic roles and keywords. Arguments to preserve are in- cluded in the context, along with blanks (<id_*>) de- noting where new generated text may be inserted. We specify desired perturbations by modifying each con- trol code (e.g., changing role LOCATIVE)TEMPORAL in (B), verb tense past)present, and patient keyword speci- ficity complete)partial). Given these perturbed control codes in the input (C), Tailor generates a new sentence (D) that reflects the desired perturbations. a sentiment style transferer …

Content truncated to 500 words