Transformer Semantic Parsing

Transformer Semantic Parsing Gabriela Ferraro1,2 and Hanna Suominen2,1,3 1Commonwealth Scientific and Industrial Research Organisation, Australia 2Research School of Computer Science, The Australian National University, Australia 3Department of Future Technologies, University of Turku, Finland gabriela.ferraro@data61.csiro.au hanna.suominen@anu.au Abstract In neural semantic parsing, sentences are mapped to meaning representations using encoder–decoder frameworks. In this paper, we propose to apply the Transformer architec- ture, instead of recurrent neural networks, to this task. Experiments in two data sets from different domains and with different levels of difficulty show that our model achieved better results than strong baselines in certain settings and competitive results across all our experi- ments. 1 Introduction Semantic parsing maps natural language sentences to meaning representations including, but not lim- ited to, logical formulas, Structured Query Lan- guage (SQL) queries, or executable codes. In re- cent years, end-to-end neural semantic parsing has achieved good results (Dong and Lapata, 2016; Jia and Liang, 2016a; Ling et al., 2016; Dong and La- pata, 2018; Finegan-Dollak et al., 2018). The main advantage of these models is that they do not re- quire intermediate representations, lexicons, man- ually designed templates, or handcrafted features. Current neural semantic parsing models use encoder–decoder architectures with Recurrent Neural Networks (RNNs). One drawback of RNNs is their inability to capture long distance relationships between input tokens or between in- put and output tokens (Yu et al., 2019). Hence, to model dependencies disregarding their distance, the best performing models also include some kind of an attention mechanism (Vaswani et al., 2017), which allows to model dependencies more accurately; specifically, the Transformer archi- tecture introduced in this last reference has be- come the new state-of-the-art for sequence-to- sequence problems. The Transformer architecture consists of a self- attention mechanism and does not include recur- rent layers. Unlike RNNs, in which sequences are processed sequentially — word by word — Transformer models process the entire sentence as a whole. This characteristic is particularly bene- ficial in capturing long distance dependencies as the self-attention mechanism sees all the words at once. Despite the success of Transformer, to the best of our knowledge, prior to Li (2019) that founded our paper, this framework has never been applied to semantic parsing before. Thus, in this paper, we propose the Transformer architecture for semantic parsing. A well-known limitation of sequence-to- sequence models is their inability to learn competitive parameter values for words that are rare in a given data set. To alleviate this problem, a common method is to anonymise entities with their respective types. For example, city names such as Denver are anonymised as ci0 and later, as part of post-processing, put back in the output utterance. Neural semantic parsing models are usually trained and tested using data sets where variables are identified and anonymised beforehand — as in the example above — which considerably reduces the difficulty of the semantic parsing task (Finegan-Dollak et al., 2018). As a result, many input sentences of the test set are seen prematurely while training. Consequently, we have a twofold approach …

Content truncated to 500 words