Learning Implicit Text Generation via Feature Matching

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3855–3863 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3855 Learning Implicit Text Generation via Feature Matching Inkit Padhi, Pierre Dognin, Ke Bai‡, Cicero Nogueira dos Santos†∗ Vijil Chenthamarakshan, Youssef Mroueh, Payel Das IBM Research, ‡Duke University, †Amazon AWS AI inkpad@ibm.com, ke.bai@duke.edu, cicnog@amazon.com {pdognin,ecvijil,mroueh,daspa}@us.ibm.com Abstract Generative feature matching network (GFMN) is an approach for training implicit genera- tive models for images by performing moment matching on features from pre-trained neu- ral networks. In this paper, we present new GFMN formulations that are effective for se- quential data. Our experimental results show the effectiveness of the proposed method, Se- qGFMN, for three distinct generation tasks in English: unconditional text generation, class- conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial ap- proaches for text generation and text style transfer. 1 Introduction Generative feature matching networks (GFMNs) (dos Santos et al., 2019) has been recently proposed for learning implicit generative models by perform- ing moment matching on features from pre-trained neural networks. This approach demonstrated that GFMN could produce state-of-the-art image gen- erators while avoiding instabilities associated with adversarial learning. Similarly to training gener- ative adversarial networks (GANs) (Goodfellow et al., 2014), GFMN training requires to backpropa- gate through the generated data to update the model parameters. This backpropagation through the gen- erated data, combined with adversarial learning instabilities, has proven to be a compelling chal- lenge when applying GANs for discrete data such as text. However, it remains unknown if this is also an issue for feature matching networks since the effectiveness of GFMN for sequential discrete data has not yet been studied. In this work, we investigate the effectiveness of GFMN for different text generation tasks. As a ∗*work done prior to joining Amazon first contribution, we propose a new formulation of GFMN for unconditional sequence generation, which we name Sequence-GFMN or SeqGFMN for short, by performing token level feature match- ing. SeqGFMN has a stable training because it does not concurrently train a discriminator, which in principle could easily learn to distinguish be- tween one-hot and soft one-hot representations. As a result, we can use soft one-hot representations that the generator outputs during training without using the Gumbel softmax or REINFORCE algo- rithm as needed in GANs for text. Additionally, different from GANs (Zhu et al., 2018), SeqGFMN can produce meaningful text without the need of pre-training the generator with maximum likeli- hood estimation (MLE). We perform experiments using Bidirectional Encoder Representations from Transformers (BERT), GloVe, and FastText as our feature extractor networks. We use two different corpora, and assess both the quality and diversity of the generated texts with three different quan- titative metrics: BLEU, Self-BLEU and Fr´echet Infersent Distance (FID). Additionally, we show that the latent space induced by SeqGFMN con- tains semantic and syntactic structure, as evidenced by interpolations in the z space. Our second contribution consists in proposing a new strategy for …

Content truncated to 500 words