Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing

Proceedings of the 29th International Conference on Computational Linguistics, pages 3164–3174 October 12–17, 2022. 3164 Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing Ran Ji and Jianmin Ji∗ University of Science and Technology of China, Hefei, China jiran@mail.ustc.edu.cn, jianmin@ustc.edu.cn Abstract Semantic parsing considers the task of map- ping a natural language sentence into a target formal representation, where various sophis- ticated sequence-to-sequence (seq2seq) mod- els have been applied with promising results. Generally, these target representations follow a syntax formalism that limits permitted forms. However, it is neither easy nor flexible to ex- plicitly integrate this syntax formalism into a neural seq2seq model. In this paper, we present a structure-aware self-attention lan- guage model to capture structural information of target representations and propose a knowl- edge distillation based approach to incorporat- ing the target language model into a seq2seq model, where grammar rules or sketches are not required in the training process. An abla- tion study shows that the proposed language model can notably improve the performance of the baseline model. The experiments show that our method achieves new state-of-the-art performance among neural approaches on four semantic parsing (ATIS, GEO) and Python code generation (Django, CoNaLa) tasks. 1 Introduction Semantic parsing aims to map a natural language sentence into a machine executable formal repre- sentation, which has been considered as one of the prime challenges nowadays in natural language pro- cessing (NLP). These target formal representations can generally be divided into three categories (Ka- math and Das, 2018), i.e., logical forms, like first order sentences or λ-calculus expressions (Zettle- moyer and Collins, 2005), programming language statements, like Python code or SQL programs, and graph-based forms, like labeled graphs in Abstract Meaning Representation (AMR) (Banarescu et al., 2013). In this paper, we focus on semantic parsing that yields logical forms. ∗The corresponding author. Target logical forms often follow a syntax for- malism that limits permitted formulas, which can be used to filter the output and improve the perfor- mance of semantic parsing. For example, in the pre- neural era, CCG based approaches (Kwiatkowski et al., 2013) achieved significant performance gains by introducing a linguistically motivated gram- mar induction scheme. Some neural semantic parsers (Yin and Neubig, 2018; Sun et al., 2020) first transduce the natural language utterance into an Abstract Syntax Tree (AST), then serve it as an intermediate meaning representation to incor- porate with grammar rules for the target logical form. Semantic parsing can also be considered as a seq2seq transduction problem, where the decoder can leverage structural features of target represen- tations. In particular, hierarchical tree decoders are applied in (Dong and Lapata, 2016; Alvarez- Melis and Jaakkola, 2017; Sun et al., 2019) to take into account the tree structure of the logical expres- sion. Decoders constrained by a grammar model are applied in (Xiao et al., 2016; Yin and Neu- big, 2017; Krishnamurthy et al., 2017; Dong and Lapata, 2018). The uncertainty-driving adaptive decoding is used to guide the decoder in (Zhang et al., 2019). Relatively …

Content truncated to 500 words