RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7716–7733 July 9-14, 2023 ©2023 Association for Computational Linguistics RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs Afra Feyza Akyürek1 Ekin Akyürek2 Ashwin Kalyan4 Peter Clark4 Derry Wijaya1,3 Niket Tandon4 1Boston University 2MIT CSAIL 3Monash University Indonesia 4Allen Institute for Artificial Intelligence Abstract Despite their unprecedented success, even the largest language models make mistakes. Sim- ilar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feed- back to guide them in repairing their outputs. Because human-generated critiques are expen- sive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream mod- els to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spa- tially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, sum- marization and alphabetization and show rela- tive improvements up to 10% in multiple text similarity metrics over other learned, retrieval- augmented or prompting-based critique genera- tors.1 1 Introduction Correcting model outputs is a pressing challenge in natural language generation (Ribeiro et al., 2018; Reid and Neubig, 2022), emerging across many use-cases such as style transfer (Mallinson et al., 1A significant portion of this work was done while Feyza was an intern at Allen AI. Correspondences to Afra Feyza Akyürek (akyurek@bu.edu). put soap in dishwasher Action Planning Topic-Based Summarization input ( ): x Passage: When did it all begin? Where it always does I suppose, with the vows […] Question: Summarize the plot of the story. model output ( ):̂y 1. open dishwasher door 2. grab some dish soap 3. put soap in dispenser Michael sits in the waiting room, thinking about his family […] he was betrayed by his father Bill […] critique ( ): c One cannot put soap in the dispenser without first opening it. Bill wasn’t Michael’s father. corrected output ( ): ̂ynew 1. open dishwasher door 2. grab some dish soap 3. open dispenser 4. put soap in dispenser Michael sits in the waiting room, thinking about his marriage […] he was betrayed by his friend Bill […] input ( ): x model output ( ):̂y critique ( ): c corrected output ( ): ̂ynew Figure 1: Two examples for action planning (Tandon et al., 2021) and summarization (Saunders et al., 2022) tasks show- case a scenario where initial predictions by a learned model (ˆy) are incorrect. Human-written critiques (c) indicate errors in model …

Content truncated to 500 words