An Invariant Learning Characterization of Controlled Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3186–3206 July 9-14, 2023 ©2023 Association for Computational Linguistics An Invariant Learning Characterization of Controlled Text Generation Carolina Zheng1∗, Claudia Shi1,2∗, Keyon Vafa1, Amir Feder1, David M. Blei1 1Columbia University 2FAR AI Abstract Controlled generation refers to the problem of creating text that contains stylistic or semantic attributes of interest. Many approaches reduce this problem to training a predictor of the desired attribute. For example, researchers hoping to deploy a large language model to produce non-toxic content may use a toxicity classifier to filter generated text. In practice, the generated text to classify, which is determined by user prompts, may come from a wide range of distributions. In this paper, we show that the performance of controlled generation may be poor if the distributions of text in response to user prompts differ from the distribution the predictor was trained on. To address this prob- lem, we cast controlled generation under dis- tribution shift as an invariant learning problem: the most effective predictor should be invariant across multiple text environments. We then dis- cuss a natural solution that arises from this char- acterization and propose heuristics for selecting natural environments. We study this charac- terization and the proposed method empirically using both synthetic and real data. Experiments demonstrate both the challenge of distribution shift in controlled generation and the potential of invariance methods in this setting. 1 Introduction The development of large language models (LLMs) has changed the landscape of research in NLP. Sim- ply by conditioning on a prompt, an LLM can pro- duce fluent and readable text. By using different and well-thought-out prompts, it can be adapted to many applications [6, 9, 35, 38, 44, 50]. But this increase in adaptability has also led to a greater need for controlled generation, to be able to generate text from an LLM that adheres to certain attributes. For example, suppose we want to use ∗ denotes equal contribution. Author order was decided by coin toss. Correspondence to: <carolina.z@columbia.edu>, <claudia.j.shi@gmail.com>. an LLM as a chatbot and deploy it to a large set of users. They might prompt the model in many differ- ent ways, such as by asking for advice, information, or just playing with its capabilities. We would like the users to freely explore the chatbot, but we also want to ensure that the text it generates is not toxic — that is, not rude, disrespectful, or unreasonable. How can we allow users to freely prompt it, but ensure that the LLM does not produce toxic text? There have been many approaches to solving this problem, each trying to ensure that the text pro- duced by a prompted LLM adheres to the attribute, e.g., that it is not toxic [10, 24, 25, 47, 53]. Here we build on the simple method of filtering. Filter- ing reduces the problem of controlled generation to one of building a good classifier of the targeted attribute. First we …

Content truncated to 500 words