Improving Adversarial Text Generation by Modeling the Distant Future

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2516–2531 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2516 Improving Adversarial Text Generation by Modeling the Distant Future Ruiyi Zhang1, Changyou Chen2, Zhe Gan3, Wenlin Wang4 Dinghan Shen3, Guoyin Wang1, Zheng Wen5, Lawrence Carin1 1 Duke University 2 University at Buffalo 3 Microsoft Dynamics 365 AI 4 Citadel LLC 5 DeepMind ryzhang@cs.duke.edu Abstract Auto-regressive text generation models usu- ally focus on local fluency, and may cause inconsistent semantic meaning in long text generation. Further, automatically generating words with similar semantics is challenging, and hand-crafted linguistic rules are difficult to apply. We consider a text planning scheme and present a model-based imitation-learning ap- proach to alleviate the aforementioned issues. Specifically, we propose a novel guider net- work to focus on the generative process over a longer horizon, which can assist next-word prediction and provide intermediate rewards for generator optimization. Extensive experi- ments demonstrate that the proposed method leads to improved performance. 1 Introduction Text generation is an important area of investiga- tion within machine learning. Recent work has shown excellent performance on a number of tasks, by combining reinforcement learning (RL) and generative models. Example applications include image captioning (Ren et al., 2017; Rennie et al., 2016), text summarization (Li et al., 2018b; Paulus et al., 2017; Rush et al., 2015), and adversarial text generation (Guo et al., 2017; Lin et al., 2017; Yu et al., 2017; Zhang et al., 2017; Zhu et al., 2018). The sequence-to-sequence framework (Seq2Seq) (Sutskever et al., 2014) is a popular technique for text generation. However, models from such a setup are typically trained to predict the next token given previous ground-truth tokens as input, caus- ing what is termed exposure bias (Ranzato et al., 2016). By contrast, sequence-level training with RL provides an effective means of solving this chal- lenge, by treating text generation as a sequential decision-making problem. By directly optimizing an evaluation score (cumulative rewards) (Ranzato et al., 2016), state-of-the-art results have been ob- tained in many text-generation tasks (Paulus et al., 2017; Rennie et al., 2016). However, one problem in such a framework is that rewards in RL training are particularly sparse, since a scalar reward is typ- ically only available after an entire sequence has been generated. Furthermore, the recurrent models focus more on local fluency, and may cause incon- sistent semantic meanings for long text generation. For RL-based text generation, most existing works rely on a model-free framework, which has been criticized for its high variance and poor sam- ple efficiency (Sutton and Barto, 1998). On the other hand, while model-based RL methods do not suffer from these issues, they are usually diffi- cult to train in complex environments. Further, a learned policy is usually restricted by the capacity of an environment model. Recent developments on model-based RL (Gu et al., 2016; Kurutach et al., 2018; Nagabandi et al., 2017) combine the advan- tages of these two approaches, and have achieved improved performance by …

Content truncated to 500 words