TILGAN: Transformer-based Implicit Latent GAN for Diverse and Coherent Text Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4844–4858 August 1–6, 2021. ©2021 Association for Computational Linguistics 4844 TILGAN: Transformer-based Implicit Latent GAN for Diverse and Coherent Text Generation Shizhe Diao♦∗, Xinwei Shen♦∗, KaShun Shum♦, Yan Song♠♥, Tong Zhang♦ ♦The Hong Kong University of Science and Technology {sdiaoaa, xshenal, ksshumab, tongzhang}@ust.hk ♠The Chinese University of Hong Kong (Shenzhen) ♥Shenzhen Research Institute of Big Data songyan@cuhk.edu.cn Abstract Conventional autoregressive models have achieved great success in text generation but suffer from the exposure bias problem in that token sequences in the training and in the gen- eration stages are mismatched. While gener- ative adversarial networks (GANs) can rem- edy this problem, existing implementations of GANs directly on discrete outputs tend to be unstable and lack diversity. In this work, we propose TILGAN, a Transformer- based Implicit Latent GAN, which combines a Transformer autoencoder and GAN in the la- tent space with a novel design and distribution matching based on the Kullback-Leibler (KL) divergence. Specifically, to improve local and global coherence, we explicitly introduce a multi-scale discriminator to capture the seman- tic information at varying scales among the sequence of hidden representations encoded by Transformer. Moreover, the decoder is en- hanced by an additional KL loss to be consis- tent with the latent-generator. Experimental re- sults on three benchmark datasets demonstrate the validity and effectiveness of our model, by obtaining significant improvements and a bet- ter quality-diversity trade-off in automatic and human evaluation for both unconditional and conditional generation tasks.1 1 Introduction In recent years, Transformer-based autoregres- sive (AR) models have made a dramatic impact in text generation tasks such as machine transla- tion (Vaswani et al., 2017; Wang et al., 2019) and dialogue systems (Le et al., 2019; Ham et al., 2020), especially with the emergence of large pre-trained language models (Radford et al., 2019; Brown et al., 2020; Wu et al., 2020). However, AR models pre- dict the next token conditioned on the ground truth *Equal Contribution. 1Our code is available at https://github.com/ shizhediao/TILGAN. during training and on its own previously generated token during inference, which leads to a mismatch between training and generation stages, and this causes low quality of generated texts and bad gen- eralization ability of models on unseen data (Wise- man and Rush, 2016; Welleck et al., 2020). Generative adversarial networks (GANs, Good- fellow et al., 2014) provide a promising approach to solve the exposure bias problem (Yu et al., 2017; Kusner and Hernández-Lobato, 2016; Zhang et al., 2017). This is because GANs aim at matching the distributions of the generated and real data instead of forcing the model output to align with the single correct sequence, and thus provide the potential to bypass the discrepancy issue. However, it is non-trivial to apply GANs to discrete data since the gradients cannot be normally back-propagated through discrete tokens. Existing approaches have implemented the adversarial discrete generation training by reinforcement learning (RL) (Yu et al., 2017; Lin et al., 2017; Guo et al., 2018; Fedus et al., …

Content truncated to 500 words