Evaluating Text GANs as Language Models

Proceedings of NAACL-HLT 2019, pages 2241–2247 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2241 Evaluating Text GANs as Language Models Guy Tevet∗,1,2 Gavriel Habib∗,1,2 Vered Shwartz3 Jonathan Berant1,4 1School of Computer Science, Tel-Aviv University 2Department of Electrical Engineering, Tel-Aviv University 3Computer Science Department, Bar-Ilan University 4Allen Institute for Artificial Intelligence {guytevet@mail,gavrielhabib@mail,joberant@cs}.tau.ac.il vered1986@gmail.com Abstract Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for under- standing the potential of GANs for text gen- eration is the lack of a clear evaluation met- ric. In this work, we propose to approxi- mate the distribution of text generated by a GAN, which permits evaluating them with tra- ditional probability-based LM metrics. We ap- ply our approximation procedure on several GAN-based models and show that they cur- rently perform substantially worse than state- of-the-art LMs. Our evaluation procedure pro- motes better understanding of the relation be- tween GANs and LMs, and can accelerate progress in GAN-based text generation. 1 Introduction Neural networks have revolutionized the field of text generation, in machine translation (Sutskever et al., 2014; Neubig, 2017; Luong et al., 2015; Chen et al., 2018), summarization (See et al., 2017), image captioning (You et al., 2016) and many other applications (Goldberg, 2017). Traditionally, text generation models are trained by going over a gold sequence of symbols (char- acters or words) from left-to-right, and maximiz- ing the probability of the next symbol given the history, namely, a language modeling (LM) ob- jective. A commonly discussed drawback of such LM-based text generation is exposure bias (Ran- zato et al., 2015): during training, the model pre- dicts the next token conditioned on the ground truth history, while at test time prediction is based on predicted tokens, causing a train-test mismatch. Models trained in this manner often struggle to overcome previous prediction errors. Generative Adversarial Networks (Goodfellow et al., 2014) offer a solution for exposure bias. ∗ The authors contributed equally Originally introduced for images, GANs leverage a discriminator, which is trained to discriminate between real images and generated images via an adversarial loss. In such a framework, the genera- tor is not directly exposed to the ground truth data, but instead learns to imitate it using global feed- back from the discriminator. This has led to sev- eral attempts to use GANs for text generation, with a generator using either a recurrent neural network (RNN) (Yu et al., 2017; Guo et al., 2017; Press et al., 2017; Rajeswar et al., 2017), or a Convo- lutional Neural Network (CNN) (Gulrajani et al., 2017; Rajeswar et al., 2017). However, evaluating GANs is more difficult than evaluating LMs. While in language model- ing, evaluation is based on the log-probability of a model on held-out text, this cannot be straight- forwardly extended to GAN-based text genera- tion, because the generator outputs discrete to- kens, rather than a probability distribution. Cur- rently, there is …

Content truncated to 500 words