On Robustness of Neural Semantic Parsers

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3333–3342 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3333 On Robustness of Neural Semantic Parsers Shuo Huang1, Zhuang Li1, Lizhen Qu1∗1, Lei Pan2 Faculty of Information and Technology, Monash University1 School of Info Technology, Deakin University2 shua0043@student.monash.edu Zhuang.Li@monash.edu Lizhen.Qu@monash.edu l.pan@deakin.edu.au Abstract Semantic parsing maps natural language (NL) utterances into logical forms (LFs), which un- derpins many advanced NLP problems. Se- mantic parsers gain performance boosts with deep neural networks, but inherit vulnerabili- ties against adversarial examples. In this pa- per, we provide the empirical study on the ro- bustness of semantic parsers in the presence of adversarial attacks. Formally, adversaries of semantic parsing are considered to be the per- turbed utterance-LF pairs, whose utterances have exactly the same meanings as the origi- nal ones. A scalable methodology is proposed to construct robustness test sets based on exist- ing benchmark corpora. Our results answered five research questions in measuring the sate- of-the-art parsers’ performance on robustness test sets, and evaluating the effect of data aug- mentation. 1 Introduction Semantic parsing aims to map natural language (NL) utterances into logical forms (LFs), which can be executed on a knowledge base (KB) to yield denotations (Kamath and Das, 2018). At the core of the state-of-the-art (SOTA) semantic parsers are deep learning models, which are widely known to be vulnerable to adversarial samples (Zhang et al., 2020). This kind of examples is created by adding tiny perturbations to inputs but can severely deteriorate model performance. To the best of our knowledge, despite the popularity of semantic parsing (Kamath and Das, 2018), there is still no published work on studying the robustness of neural semantic parsers against adversarial ex- amples. Therefore, we conduct the first empirical study to evaluate the effect of adversarial examples on SOTA neural semantic parsers. ∗corresponding author Unlike other disciplines, it is unclear what ad- versaries are for semantic parsers. For computer vision systems, adversaries are often generated by modifying inputs with imperceptible perturba- tions. In contrast, a flip of single word or a char- acter in an utterance can significantly change its meaning so that the changes are perceptible by hu- mans. To address this issue, (Michel et al., 2019) argue that adversaries for sequence-to-sequence models should maximally retain meanings af- ter perturbing inputs. However, any meaning- changing utterances are supposed to have differ- ent meaning representations. A robust semantic parser should be invariant to meaning-preserving modifications. In light of this, given a semantic parser, we define its adversaries as the perturbed utterances satisfying two conditions: i) they have exactly the same meanings as the original ones ac- cording to human judgements; and ii) the parser consistently produces incorrect LFs on them. Although new evaluation frameworks are pro- posed for NLP tasks (Xu et al., 2020; Michel et al., 2019), there is no framework designed for assessing robustness of semantic parsers against meaning-preserving adversaries. The current eval- uation metrics focus only …

Content truncated to 500 words