Semantic Parsing with Semi-Supervised Sequential Autoencoders

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1078–1087, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Semantic Parsing with Semi-Supervised Sequential Autoencoders Tom´aˇs Koˇcisk´y†‡ G´abor Melis† Edward Grefenstette† Chris Dyer† Wang Ling† Phil Blunsom†‡ Karl Moritz Hermann† †Google DeepMind ‡University of Oxford {tkocisky,melisgl,etg,cdyer,lingwang,pblunsom,kmh}@google.com Abstract We present a novel semi-supervised approach for sequence transduction and apply it to se- mantic parsing. The unsupervised component is based on a generative model in which latent sentences generate the unpaired logical forms. We apply this method to a number of semantic parsing tasks focusing on domains with lim- ited access to labelled training data and ex- tend those datasets with synthetically gener- ated logical forms. 1 Introduction Neural approaches, in particular attention-based sequence-to-sequence models, have shown great promise and obtained state-of-the-art performance for sequence transduction tasks including machine translation (Bahdanau et al., 2015), syntactic con- stituency parsing (Vinyals et al., 2015), and seman- tic role labelling (Zhou and Xu, 2015). A key re- quirement for effectively training such models is an abundance of supervised data. In this paper we focus on learning mappings from input sequences x to output sequences y in domains where the latter are easily obtained, but annotation in the form of (x, y) pairs is sparse or expensive to produce, and propose a novel architecture that ac- commodates semi-supervised training on sequence transduction tasks. To this end, we augment the transduction objective (x �→ y) with an autoencod- ing objective where the input sequence is treated as a latent variable (y �→ x �→ y), enabling training from both labelled pairs and unpaired output sequences. This is common in situations where we encode nat- ural language into a logical form governed by some grammar or database. While such an autoencoder could in principle be constructed by stacking two sequence transduc- ers, modelling the latent variable as a series of dis- crete symbols drawn from multinomial distributions creates serious computational challenges, as it re- quires marginalising over the space of latent se- quences Σ∗ x. To avoid this intractable marginalisa- tion, we introduce a novel differentiable alternative for draws from a softmax which can be used with the reparametrisation trick of Kingma and Welling (2014). Rather than drawing a discrete symbol in Σx from a softmax, we draw a distribution over sym- bols from a logistic-normal distribution at each time step. These serve as continuous relaxations of dis- crete samples, providing a differentiable estimator of the expected reconstruction log likelihood. We demonstrate the effectiveness of our proposed model on three semantic parsing tasks: the GEO- QUERY benchmark (Zelle and Mooney, 1996; Wong and Mooney, 2006), the SAIL maze navigation task (MacMahon et al., 2006) and the Natural Language Querying corpus (Haas and Riezler, 2016) on Open- StreetMap. As part of our evaluation, we introduce simple mechanisms for generating large amounts of unsupervised training data for two of these tasks. In most settings, the semi-supervised model out- performs the supervised model, both when trained …

Content truncated to 500 words