The CRINGE Loss: Learning what language not to model

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8854–8874 July 9-14, 2023 ©2023 Association for Computational Linguistics The CRINGE Loss: Learning what language not to model Leonard Adolphs Meta AI & ETH Zürich Tianyu Gao Meta AI & Princeton University Jing Xu Meta AI Kurt Shuster Meta AI Sainbayar Sukhbaatar Meta AI Jason Weston Meta AI Abstract Standard language model training employs gold human documents or human-human in- teraction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be allevi- ated with relatively small amounts of negative data – examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models out- perform multiple strong baselines and are con- ceptually simple, easy to train and implement. 1 Introduction Through the rise of large Transformers (Vaswani et al., 2017), both language models (Brown et al., 2020; Chowdhery et al., 2022) and conversational agents (Shuster et al., 2022) have become much more powerful in recent years – up to the point that it is possible to engage with them in useful and non- trivial interactions. However, employing standard language model training and scaling the model size and amount of training data fails to resolve a num- ber of issues. In particular, models can still suffer from toxicity and bias (Gehman et al., 2020), lack of (long-term) coherence (Nie et al., 2020) or fail to address user’s intent (Ouyang et al., 2022b). A growing body of work is instead investigating ways to train models beyond the standard language mod- eling objective, given access to examples of such failure cases, by incorporating this information into the training objective (Welleck et al., 2020; Krause et al., 2020; Yang and Klein, 2021; Nakano et al., 2021; Askell et al., 2021; Arora et al., 2022). In this work, we study the setting where the train- ing set involves a given set of positive example se- quences, as is commonly used for language model Figure 1: The CRINGE loss works by penalizing the output sequence of negative examples (shown in red). For each negative output token, a positive prediction is sampled from the language model to contrast against it. Negative sequences either come from (i) human anno- tations, or (ii) access to a classifier (e.g., trained from the human annotations) that can be used to iteratively la- bel the model’s own generations and apply the CRINGE loss to those examples as well. Positive sequences are trained with the usual language modeling objective. training, and a set of negative example sequences, which are completions given a prompt that a model should not generate. We propose a new learning method, the CRINGE (ContRastive …

Content truncated to 500 words