A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2344–2354 August 1–6, 2021. ©2021 Association for Computational Linguistics 2344 A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples Yuxuan Wang∗1 Wanxiang Che1 Ivan Titov2,3 Shay B. Cohen2 Zhilin Lei1 Ting Liu1 1Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology 2ILCC, School of Informatics, University of Edinburgh 3ILLC, University of Amsterdam {yxwang, car, zllei, tliu}@ir.hit.edu.cn {ititov, scohen}@inf.ed.ac.uk Abstract Previous work on adversarial attacks on depen- dency parsers has mostly focused on attack methods, as opposed to the quality of adver- sarial examples, which in previous work has been relatively low. To address this gap, we propose a method to generate high-quality ad- versarial examples with a higher number of candidate generators and stricter filters, and then verify their quality using automatic and human evaluations. We perform analysis with different parsing models and observe that: (i) injecting words not used in the training stage is an effective attack strategy; (ii) adversarial examples generated against a parser strongly depend on the parser model, the token embed- dings, and even the specific instantiation of the model (i.e., a random seed). We use these insights to improve the robustness of English parsing models, relying on adversarial training and model ensembling.1 1 Introduction Neural network-based models have achieved great successes in a wide range of NLP tasks. However, recent work has shown that their performance can be easily undermined with adversarial examples that would pose no confusion for humans (Zhang et al., 2020). As an increasing number of successful adversarial attackers have been developed for NLP tasks, the quality of the adversarial examples they generate has been questioned (Morris et al., 2020). The definition of a valid successful adversar- ial example differs across target tasks. In seman- tic tasks such as sentiment analysis (Zhang et al., 2019) and textual entailment (Jin et al., 2020), a valid successful adversarial example needs to be able to alter the prediction of the target model while burgh.∗ Work partially done while at the University of Edin- 1Our code is available at: https://github.com/ WangYuxuan93/DepAttacker.git preserving the semantic content and fluency of the original text. In contrast, in the less explored field of attacking syntactic tasks, the syntactic structure, rather than the semantic content, must be preserved while also maintaining the fluency. Preserving the syntactic structure enables us to use the gold syn- tactic structure of the original sentence in the evalu- ation process. While preserving the fluency ensures that ungrammatical adversarial examples, which not only fool the target model but also confuse hu- mans, will not be considered valid. Therefore in this paper, we evaluate the quality of an adversarial example in two aspects, namely the fluency and syntactic structure preservation. Recently, Zheng et al. (2020) proposed the first dependency parser attacking algorithm based on word-substitution which depended entirely on BERT (Devlin et al., 2019) to generate candidate substitutes. The rational was that the use of the pre-trained language model will …

Content truncated to 500 words