A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5090–5099, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5090 A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features Hongyin Tang1,2, Miao Li1,2, Beihong Jin1,2,∗ 1State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing China {tanghongyin14,limiao17}@otcaix.iscas.ac.cn Beihong@iscas.ac.cn Abstract Text generation is among the most fundamen- tal tasks in natural language processing. In this paper, we propose a text generation model that learns semantics and structural features si- multaneously. This model captures structural features by a sequential variational autoen- coder component and leverages a topic mod- eling component based on a Gaussian distri- bution to enhance the recognition of text se- mantics. To make the reconstructed text more coherent to the topics, the model further adapts the encoder of the topic modeling component for a discriminator. The results of experi- ments over several datasets demonstrate that our model outperforms several states of the art models in terms of text perplexity and topic co- herence. Moreover, the latent representations learned by our model is superior to others in a text classification task. Finally, given the in- put texts, our model can generate meaningful texts which hold similar structures but under different topics. 1 Introduction Text generation is a fundamental task in natu- ral language processing (NLP). Existing methods for text generation are mostly limited in super- vised setting and designed for specific applications (e.g., machine translation (Bahdanau et al., 2015), text summarization (Rush et al., 2015)). Sev- eral research work (Yu et al., 2017; Zhang et al., 2016) attempts generic text generation based on deep generative models (e.g., GAN, VAE). How- ever, models based on GAN(Generative Adver- sarial Network) are not able to generate explicit latent codes with salient features of texts. Dif- ferent from GAN-based models, the VAE (Varia- tional Autoencoder) and its variants can get latent codes of texts with reconstructing texts by its de- coder, where it is assumed that the generation pro- cess is controlled by codes in a continuous latent space. A natural way to implement the VAEs is to adopt autoregressive networks (e.g., RNNs) as the decoder and encoder. This kind of implementa- tion of VAEs considers sequential information of texts, and is able to model the linguistic structure of texts. But it is not good at modeling seman- tics (Dieng et al., 2017) and may cause the latent variable collapse problem (Bowman et al., 2016) which leads that the decoder ignores information from the inferred latent codes. In general, texts inherently contain semantic features and structural features. Only when the latent codes of texts contain structural and seman- tic information can high-quality texts be generated from the codes. As far as the standard VAE is con- cerned, it assumes that the latent code is Gaussian distributed so that we cannot distinguish which part of …

Content truncated to 500 words