Syntax-Infused Variational Autoencoder for Text Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2069–2078 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2069 Syntax-Infused Variational Autoencoder for Text Generation Xinyuan Zhang1∗, Yi Yang2∗, Siyang Yuan1, Dinghan Shen1, Lawrence Carin1 1Duke University 2ASAPP Inc. xy.zhang@duke.edu, yyang@asapp.com Abstract We present a syntax-infused variational au- toencoder (SIVAE), that integrates sentences with their syntactic trees to improve the gram- mar of generated sentences. Distinct from existing VAE-based text generative models, SIVAE contains two separate latent spaces, for sentences and syntactic trees. The ev- idence lower bound objective is redesigned correspondingly, by optimizing a joint distri- bution that accommodates two encoders and two decoders. SIVAE works with long short- term memory architectures to simultaneously generate sentences and syntactic trees. Two versions of SIVAE are proposed: one cap- tures the dependencies between the latent vari- ables through a conditional prior network, and the other treats the latent variables indepen- dently such that syntactically-controlled sen- tence generation can be performed. Experi- mental results demonstrate the generative su- periority of SIVAE on both reconstruction and targeted syntactic evaluations. Finally, we show that the proposed models can be used for unsupervised paraphrasing given different syn- tactic tree templates. 1 Introduction Neural language models based on recurrent neural networks (Mikolov et al., 2010) and sequence-to- sequence architectures (Sutskever et al., 2014) have revolutionized the NLP world. Deep latent variable modes, in particular, the variational autoencoders (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) integrating inference models with neural lan- guage models have been widely adopted on text generation (Bowman et al., 2016; Yang et al., 2017; Kim et al., 2018), where the encoder and the de- coder are modeled by long short-term memory ∗Part of this work was done when the first two authors were at Bloomberg. S NP NP DT The NN book SBAR WHNP IN that S NP PRP you VP VBP love VP VBZ is ADJP JJ good . . Figure 1: An example of a constituency tree structure. (LSTM) networks (Chung et al., 2014). For a ran- dom vector from the latent space representing an unseen input, the decoder can generate realistic- looking novel data in the context of a text model, making the VAE an attractive generative model. Compared to simple neural language models, the latent representation in a VAE is supposed to give the model more expressive capacity. Although syntactic properties can be implicitly discovered by such generative models, Shi et al. (2016) show that many deep structural details are still missing in the generated text. As a result of the absence of explicit syntactic information, generative models often produce ungrammatical sentences. To address this problem, recent works attempt to leverage explicit syntactic knowledge to improve the quality of machine translation (Eriguchi et al., 2016; Bastings et al., 2017; Chen et al., 2017) and achieve good results. Motivated by such success, we suggest that deep latent variable models for text generation can also benefit from the incorporation of …

Content truncated to 500 words