Adversarial Example Generation with Syntactically Controlled Paraphrase Networks

Proceedings of NAACL-HLT 2018, pages 1875–1885 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Adversarial Example Generation with Syntactically Controlled Paraphrase Networks Mohit Iyyer⋆†‡ John Wieting⋆♣ Kevin Gimpel♦ Luke Zettlemoyer♠ Allen Institute of Artificial Intelligence† UMass Amherst‡ Carnegie Mellon University♣ Toyota Technological Institute at Chicago♦ University of Washington♠ miyyer@cs.umass.edu jwieting@cs.cmu.edu kgimpel@ttic.edu lsz@cs.washington.edu Abstract We propose syntactically controlled para- phrase networks (SCPNs) and use them to generate adversarial examples. Given a sen- tence and a target syntactic form (e.g., a con- stituency parse), SCPNs are trained to pro- duce a paraphrase of the sentence with the de- sired syntax. We show it is possible to create training data for this task by first doing back- translation at a very large scale, and then us- ing a parser to label the syntactic transforma- tions that naturally occur during this process. Such data allows us to train a neural encoder- decoder model with extra inputs to specify the target syntax. A combination of automated and human evaluations show that SCPNs gen- erate paraphrases that follow their target spec- ifications without decreasing paraphrase qual- ity when compared to baseline (uncontrolled) paraphrase systems. Furthermore, they are more capable of generating syntactically ad- versarial examples that both (1) “fool” pre- trained models and (2) improve the robustness of these models to syntactic variation when used to augment their training data. 1 Introduction Natural language processing datasets often suf- fer from a dearth of linguistic variation, which can hurt the generalization of models trained on them. Recent work has shown it is possible to easily “break” many learned models by evaluating them on adversarial examples (Goodfellow et al., 2015), which are generated by manually intro- ducing lexical, pragmatic, and syntactic variation not seen in the training set (Ettinger et al., 2017). Robustness to such adversarial examples can po- tentially be improved by augmenting the training data, as shown by prior work that introduces rule- based lexical substitutions (Jia and Liang, 2017; ⋆Authors contributed equally. I’d have to say the star and director are the big problems here negative The man is standing in the water at the base of a waterfall entailment A man is standing in the water at the base of a waterfall By the way, you know, the star and director are the big problems positive The man, at the base of the waterfall, is standing in the water A man is standing in the water at the base of a waterfall neutral S PP PRN NP VP SCPN S NP , PP , VP SCPN Figure 1: Adversarial examples for sentiment analysis (left) and textual entailment (right) generated by our syntactically controlled paraphrase network (SCPN) ac- cording to provided parse templates. In both cases, a pretrained classifier correctly predicts the label of the original sentence but not the corresponding paraphrase. Liang et al., 2017). However, more complex trans- formations, such as generating syntactically ad- versarial examples, remain an open challenge, as input semantics must be preserved in the …

Content truncated to 500 words