Generating Sentences from Disentangled Syntactic and Semantic Spaces

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6008–6019 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6008 Generating Sentences from Disentangled Syntactic and Semantic Spaces Yu Bao1∗ Hao Zhou2∗ Shujian Huang1† Lei Li2 Lili Mou3 Olga Vechtomova3 Xinyu Dai1 Jiajun Chen1 1National Key Laboratory for Novel Software Technology, Nanjing University, China 2ByteDance AI Lab, Beijing, China 3University of Waterloo, Canada {baoy,huangsj,dxy,chenjj}@nlp.nju.edu.cn {zhouhao.nlp,lileilab}@bytedance.com doublepower.mou@gmail.com, ovechtomova@uwaterloo.ca Abstract Variational auto-encoders (VAEs) are widely used in natural language generation due to the regularization of the latent space. How- ever, generating sentences from the continu- ous latent space does not explicitly model the syntactic information. In this paper, we pro- pose to generate sentences from disentangled syntactic and semantic spaces. Our proposed method explicitly models syntactic informa- tion in the VAE’s latent space by using the lin- earized tree sequence, leading to better perfor- mance of language generation. Additionally, the advantage of sampling in the disentangled syntactic and semantic latent spaces enables us to perform novel applications, such as the un- supervised paraphrase generation and syntax- transfer generation. Experimental results show that our proposed model achieves similar or better performance in various tasks, compared with state-of-the-art related work. ‡ 1 Introduction Variational auto-encoders (VAEs, Kingma and Welling, 2014) are widely used in language gen- eration tasks (Serban et al., 2017; Kusner et al., 2017; Semeniuta et al., 2017; Li et al., 2018b). VAE encodes a sentence into a probabilistic latent space, from which it learns to decode the same sentence. In addition to traditional reconstruc- tion loss of an autoencoder, VAE employs an ex- tra regularization term, penalizing the Kullback– Leibler (KL) divergence between the encoded pos- terior distribution and its prior. This property en- ables us to sample and generate sentences from the continuous latent space. Additionally, we can ∗Equal contributions. †Corresponding author. ‡We release the implementation and models at https:// github.com/baoy-nlp/DSS-VAE even manually manipulate the latent space, inspir- ing various applications such as sentence interpo- lation (Bowman et al., 2016) and text style trans- fer (Hu et al., 2017). However, the continuous latent space of VAE blends syntactic and semantic information to- gether, without modeling the syntax explicitly. We argue that it may be not necessarily the best in the text generation scenario. Recently, researchers have shown that explicitly syntactic modeling improves the generation quality in sequence-to- sequence models (Eriguchi et al., 2016; Zhou et al., 2017; Li et al., 2017; Chen et al., 2017). It is straightforward to adopt such idea in the VAE set- ting, since a vanilla VAE does not explicitly model the syntax. A line of studies (Kusner et al., 2017; G´omez-Bombarelli et al., 2018; Dai et al., 2018) propose to impose context-free grammars (CFGs) as hard constraints in the VAE decoder, so that they could generate syntactically valid outputs of programs, molecules, etc. However, the above approaches cannot be ap- plied to syntactic modeling in VAE’s continuous latent space, and thus, we do not enjoy the two …

Content truncated to 500 words