DORB: Dynamically Optimizing Multiple Rewards with Bandits

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7766–7780, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7766 DORB: Dynamically Optimizing Multiple Rewards with Bandits Ramakanth Pasunuru Han Guo Mohit Bansal UNC Chapel Hill {ram, hanguo, mbansal}@cs.unc.edu Abstract Policy gradients-based reinforcement learning has proven to be a promising approach for di- rectly optimizing non-differentiable evaluation metrics for language generation tasks. How- ever, optimizing for a specific metric reward leads to improvements in mostly that met- ric only, suggesting that the model is gam- ing the formulation of that metric in a partic- ular way without often achieving real quali- tative improvements. Hence, it is more ben- eficial to make the model optimize multiple diverse metric rewards jointly. While appeal- ing, this is challenging because one needs to manually decide the importance and scal- ing weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit ap- proach (DORB), where at each round, the ban- dit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierar- chical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our ap- proaches via various automatic metrics and hu- man evaluation on two important NLG tasks: question generation and data-to-text genera- tion. Finally, we present interpretable analyses of the learned bandit curriculum over the opti- mized rewards. 1 Introduction Recent advancements in end-to-end neural networks-based approaches have shown wide success in various sequence generation tasks: machine translation (Sutskever et al., 2014; Luong et al., 2015), dialogue systems (Vinyals and Le, 2015; Serban et al., 2016), textual summariza- tion (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017), image/video captioning (Bahdanau et al., 2015; Venugopalan et al., 2015; Pasunuru and Bansal, 2017a), question generation (Du et al., 2017; Du and Cardie, 2018; Zhang and Bansal, 2019), etc. In all of these tasks, cross-entropy loss optimization has been widely used as a standard optimization approach (Sutskever et al., 2014), but this approach suffers from exposure-bias issue (Ranzato et al., 2016) and does not optimize for the non-differentiable automatic evaluation metrics that measure the quality of the gener- ated sequence. Recent introduction of policy gradient-based reinforcement learning approaches address these issues for sequence generation tasks by directly optimizing the non-differentiable evaluation metrics (Zaremba and Sutskever, 2015; Ranzato et al., 2016; Rennie et al., 2017). However, optimizing for a particular met- ric/reward via policy gradient-based approaches often leads to improvement in mostly that specific metric, suggesting that this approach is gaming the metrics (Paulus et al., 2018). The weighted average of multiple metrics or surrogate rewards have been explored (Liu et al., 2017), but these approaches have to deal with finding the optimal scale …

Content truncated to 500 words