Reward Gaming in Conditional Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4746–4763 July 9-14, 2023 ©2023 Association for Computational Linguistics Reward Gaming in Conditional Text Generation Richard Yuanzhe Pang1 Vishakh Padmakumar1 Thibault Sellam2 Ankur P. Parikh2 He He1 1New York University 2Google DeepMind yzpang@nyu.edu Abstract To align conditional text generation model out- puts with desired behaviors, there has been an increasing focus on training the model us- ing reinforcement learning (RL) with reward functions learned from human annotations. Un- der this framework, we identify three common cases where high rewards are incorrectly as- signed to undesirable patterns: noise-induced spurious correlation, naturally occurring spuri- ous correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable pat- terns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation ex- amples and discuss potential fixes and areas for future work. 1 Introduction Natural language generation aims to automatically produce text that is fluent, relevant, and factual. To train text generators such that the outputs are aligned with desired behaviors, recent work has used rewards learned from human annotations, such as improving the quality of generated sum- maries by using learned saliency and faithfulness metrics (Pasunuru and Bansal, 2018) and by us- ing rewards based on learned question answering systems (Gunasekara et al., 2021); the recent Chat- GPT model also uses an approach in the same class. In general, this class of methods (1) collects a hu- man annotation dataset Dreward consisting of, e.g., direct ratings of generations (Sellam et al., 2020; Nakatani et al., 2022; Ramamurthy et al., 2023), la- bels of error spans in the generations (Freitag et al., 2020; Amrhein and Sennrich, 2022), or pairwise comparison of generations given the same source sequence (Stiennon et al., 2020; Wu et al., 2021; Bai et al., 2022a); (2) learns a proxy reward func- tion that scores generations (as opposed to a true reward function which is often given by human judgment) on Dreward; and then (3) learns the text generator on a dataset Dtask, using RL with the learned reward function. What could go wrong when we obtain the reward signal from humans? The rewards would rarely be robust. When training the text generator, the dis- tribution induced by the policy (i.e., the generator) changes because we frequently update it, which opens up opportunities for exploiting errors in the reward. Thus, even if the reward function performs well on the dev/test split of Dreward as an evaluator, the reward can still be gamed during RL training of the generator. Reward gaming commonly refers to the issue that when the proxy reward increases, the true reward decreases or stays stable (Amodei et …

Content truncated to 500 words