Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation

Proceedings of NAACL-HLT 2019, pages 2248–2258 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2248 Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation Md. Akmal Haidar, Mehdi Rezagholizadeh, Alan Do-Omri and Ahmad Rashid Huawei Noah’s Ark Lab, Montreal Research Center, Canada md.akmal.haidar@huawei.com, mehdi.rezagholizadeh@huawei.com alan.do.omri@huawei.com, ahmad.rashid@huawei.com Abstract Text generation with generative adversarial networks (GANs) can be divided into the text- based and code-based categories according to the type of signals used for discrimination. In this work, we introduce a novel text-based ap- proach called Soft-GAN to effectively exploit GAN setup for text generation. We demon- strate how autoencoders (AEs) can be used for providing a continuous representation of sentences, which we will refer to as soft- text. This soft representation will be used in GAN discrimination to synthesize similar soft- texts. We also propose hybrid latent code and text-based GAN (LATEXT-GAN ) approaches with one or more discriminators, in which a combination of the latent code and the soft- text is used for GAN discriminations. We per- form a number of subjective and objective ex- periments on two well-known datasets (SNLI and Image COCO) to validate our techniques. We discuss the results using several evalua- tion metrics and show that the proposed tech- niques outperform the traditional GAN-based text-generation methods. 1 Introduction Text generation is an active research area and has many real-world applications, including, but not limited to, machine translation (Bahdanau et al., 2014), AI chat bots (Li et al., 2017), im- age captioning (Xu et al., 2014), question an- swering and information retrieval (Wang et al., 2017). Recurrent neural network language mod- els (RNNLMs) (Mikolov et al., 2010) is the most popular approach for text generation which rely on maximum likelihood estimation (MLE) solu- tions such as teacher forcing (Williams and Zipser, 1989) (i.e. the model is trained to predict the next word given all the previous predicted words); however, it is well-known in the literature that MLE is a simplistic objective for this complex NLP task (Li et al., 2017). It is reported that MLE- based methods suffer from exposure bias (Huszr, 2015), which means that at training time the model is exposed to gold data only, but at test time it ob- serves its own predictions. Hence, wrong predic- tions quickly accumulate and result in poor text generation quality. However, generative adversarial networks (GANs) (Goodfellow et al., 2014) which are based on an adversarial loss function suffers less from the mentioned problems of the MLE solutions. The great success of GANs in image generation framework (Salimans et al., 2016) motivated researchers to apply its framework to NLP applications as well. GANs have been exten- sively used recently in various NLP applications such as machine translation (Wu et al., 2017; Yang et al., 2017a), dialogue models (Li et al., 2017), question answering (Yang et al., 2017b), and natural language generation (Gulrajani et al., 2017; Rajeswar et al., 2017; Press et al., 2017; Kim et al., 2017; Zhang et al., 2017; Cifka …

Content truncated to 500 words