Discourse-Aware Neural Rewards for Coherent Text Generation

Proceedings of NAACL-HLT 2018, pages 173–184 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Discourse-Aware Neural Rewards for Coherent Text Generation Antoine Bosselut1∗, Asli Celikyilmaz2, Xiaodong He3, Jianfeng Gao2, Po-Sen Huang2 and Yejin Choi1,4 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Microsoft Research 3JD AI Research 4Allen Institute for Artificial Intelligence {antoineb,yejin}@cs.washington.edu {xiaodong.he}@jd.com {aslicel,jfgao,pshuang}@microsoft.com Abstract In this paper, we investigate the use of discourse-aware rewards with reinforce- ment learning to guide a model to gen- erate long, coherent text. In particular, we propose to learn neural rewards to model cross-sentence ordering as a means to approximate desired discourse struc- ture. Empirical results demonstrate that a generator trained with the learned reward produces more coherent and less repeti- tive text than models trained with cross- entropy or with reinforcement learning with commonly used scores as rewards. 1 Introduction Defining an ideal loss for training text genera- tion models remains an open research question. Many existing approaches based on variants of re- current neural networks (Hochreiter and Schmid- huber, 1997; Cho et al., 2014) are trained using cross-entropy loss (Bahdanau et al., 2015; Vinyals et al., 2015; Xu et al., 2015; Rush et al., 2015), of- ten augmented with additional terms for topic cov- erage or task-specific supervision (Kiddon et al., 2016; Yang et al., 2017). Training with cross-entropy, however, does not always correlate well with achieving high scores on commonly used evaluation measures such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), or CIDEr (Vedantam et al., 2015). Another current line of research therefore explores train- ing generation models that directly optimize the target evaluation measure (Wu et al., 2016; Ran- zato et al., 2015; Paulus et al., 2018; Rennie et al., 2017) using reinforcement learning methods such as the REINFORCE algorithm (Williams, 1992). ∗ Work done while author was at Microsoft Research Model Teacher Reward Wash the tomatoes and cut them length-wise. Set on plate. Slice the mozzarella and put on tomatoes. Add dressing and serve cold. Generated Recipe: Gold Recipe Figure 1: The generator is rewarded for imitating the discourse structure of the gold sequence. Importantly, most automatic measures are based on local n-gram patterns, providing only a lim- ited and myopic perspective of overall text qual- ity. As a result, while models trained to directly optimize these measures can yield improvements on the same measures, they may not lead to bet- ter quality in terms of overall coherence or dis- course structure. Indeed, recent studies have re- ported cases where commonly used measures do not align well with desired aspects of generation quality (Rennie et al., 2017; Li et al., 2016). The challenge, however, is to define a global score that can measure the complex aspects of text quality beyond local n-gram patterns. In this pa- per, we investigate learning neural rewards and their use in a reinforcement learning regime with a specific focus on learning more discourse-aware and coherent text generation. Our approach shares the spirit …

Content truncated to 500 words