Domain Adaptation for Low-Resource Neural Semantic Parsing

Domain Adaptation for Low-Resource Neural Semantic Parsing Alvin Kennardi1, Gabriela Ferraro1,2, and Qing Wang1 1Research School of Computer Science, Australian National University 2Data61, CSIRO {alvin.kennardi, qing.wang}@anu.edu.au gabriela.ferraro@data61.csiro.au Abstract One key challenge for building a semantic parser in new domains is the difficulty to an- notate new datasets. In this paper, we pro- pose a sequential transfer learning method as a domain adaptation method to tackle this is- sue. We show that we can obtain a model with better generalisation on a small dataset by transferring network parameters from a model trained with a bigger dataset with sim- ilar meaning representations. We evaluate our model with different datasets as well as ver- sions of the datasets with different difficulty levels. 1 Introduction Semantic parsing maps natural language sentences into meaning representations, for example, logical formulae, SQL queries, or executable codes. The successful implementation of the encoder-decoder architecture in the machine translation (Kalch- brenner and Blunsom, 2013; Sutskever et al., 2014) has driven researchers to apply this model into semantic parsing task (Dong and Lapata, 2016; Jia and Liang, 2016; Ling et al., 2016; Dong and Lapata, 2018). These neural semantic parsing models have achieved impressive results. Semantic parsing datasets are usually domain and meaning representation dependent, thus mak- ing it difficult to re-use existing datasets for build- ing general semantic parsers or semantic parsers in new domains. The process of annotating sentences with their meaning representations for modeling new domains or augmenting the existing datasets is expensive. Prior works proposed several strate- gies to tackle this issue, such as paraphrasing (Su and Yan, 2017), decoupling structure and lexicon (Herzig and Berant, 2018), and multi-task learning (Susanto and Lu, 2017; Herzig and Berant, 2017). Our method aims to provide an alternative to the previous work. We perform transfer learning by training a model for one task using a dataset and fine-tuning the model using another related dataset. The idea of transfer learning is to utilize features, weights, or other knowledge acquired for one task to solve another related task. It has been extensively used for domain adaptation and building models to solve problems where only limited data is available (Pan and Yang, 2010). The fine-tuning transfer learning procedure has been successfully implemented in the encoder- decoder architecture for Neural Machine Transla- tion Task (NMT) (Luong and Manning, 2015; Sen- nrich et al., 2016; Servan et al., 2016). In contrast with the multi-task learning, which jointly trains several tasks together, we perform transfer learn- ing by training the first and second tasks in se- quence. Compared to models without transfer learn- ing, our experiments shows that transfer learn- ing gives a good prior for models trained with small datasets, hence improving model perfor- mance when only limited amounts of data are available. Neural semantic parsing models are usually trained and tested using datasets in which vari- ables are identified and anonymised before hand, thus considerably reducing the difficulty of the se- mantic parsing task (Finegan-Dollak et al., 2018). In this work, we use …

Content truncated to 500 words