ARAML: A Stable Adversarial Training Framework for Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4271–4281, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4271 ARAML: A Stable Adversarial Training Framework for Text Generation Pei Ke∗, Fei Huang∗, Minlie Huang†, Xiaoyan Zhu Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China kepei1106@outlook.com, f-huang18@mails.tsinghua.edu.cn aihuang@tsinghua.edu.cn, zxy-dcs@tsinghua.edu.cn Abstract Most of the existing generative adversarial net- works (GAN) for text generation suffer from the instability of reinforcement learning train- ing algorithms such as policy gradient, leading to unstable performance. To tackle this prob- lem, we propose a novel framework called Ad- versarial Reward Augmented Maximum Like- lihood (ARAML). During adversarial train- ing, the discriminator assigns rewards to sam- ples which are acquired from a stationary dis- tribution near the data rather than the gen- erator’s distribution. The generator is opti- mized with maximum likelihood estimation augmented by the discriminator’s rewards in- stead of policy gradient. Experiments show that our model can outperform state-of-the-art text GANs with a more stable training process. 1 Introduction Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models (Bengio et al., 2003; Mikolov et al., 2010). Typical frameworks such as sequence-to-sequence (seq2seq) have been ap- plied to various generation tasks, including ma- chine translation (Sutskever et al., 2014) and di- alogue generation (Vinyals and Le, 2015). The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceed- ing context (Graves, 2013). Although widely used, MLE suffers from the exposure bias problem (Bengio et al., 2015; Ran- zato et al., 2016): during test, the model sequen- tially predicts the next word conditioned on its pre- vious generated words while during training con- ditioned on ground-truth words. To tackle this ∗ Equal contribution † Corresponding author: Minlie Huang problem, generative adversarial networks (GAN) with reinforcement learning (RL) training ap- proaches have been introduced to text generation tasks (Yu et al., 2017; Che et al., 2017; Lin et al., 2017; Fedus et al., 2018; Guo et al., 2018; Shi et al., 2018; Xu et al., 2018), where the discrim- inator is trained to distinguish real and generated text samples to provide reward signals for the gen- erator, and the generator is optimized via policy gradient (Yu et al., 2017). However, recent studies have shown that poten- tial issues of training GANs on discrete data are more severe than exposure bias (Semeniuta1 et al., 2018; Caccia et al., 2018). One of the fundamental issues when generating discrete text samples with GANs is training instability. Updating the gener- ator with policy gradient always leads to an un- stable training process because it’s difficult for the generator to derive positive and stable reward …

Content truncated to 500 words