Director: Generator-Classifiers For Supervised Language Modeling

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 512–526 November 20–23, 2022. ©2022 Association for Computational Linguistics 512 DIRECTOR: Generator-Classifiers For Supervised Language Modeling Kushal Arora∗ McGill University Mila Kurt Shuster Meta AI Sainbayar Sukhbaatar Meta AI Jason Weston Meta AI Abstract Current language models achieve low perplex- ity but their resulting generations still suffer from toxic responses, repetitiveness, and con- tradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, DIRECTOR, that consists of a unified generator-classifier with both a language modeling and a classifi- cation head for each output token. Training is conducted jointly using both standard lan- guage modeling data, and data labeled with desirable and undesirable sequences. Experi- ments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, avoiding undesirable behaviors while maintaining generation qual- ity. It also outperforms existing model-guiding approaches in terms of both accuracy and effi- ciency. Our code is made publicly available1. 1 Introduction Language models are becoming a powerful tool in various machine learning applications due to recent advancements in large-scale transformer models (Brown et al., 2020). Standard language model training relies on maximizing log-likelihood over large training corpora yielding low perplexity next- token predictions. However, the resulting model generations still suffer from a number of problems. Biases may be amplified from those already present in the large training corpora, and toxic or otherwise unsafe language can be generated (Gehman et al., 2020; Welbl et al., 2021). Current models do not appear to adequately understand the deeper mean- ing of their generations and frequently contradict themselves (Nie et al., 2020). They are also known to produce repetitive text (Holtzman et al., 2019). If one has access to data labeled with such sequence generation errors, there is also no way to use it in ∗Work done during an internship at Meta AI. 1https://parl.ai/projects/director Class Head Decoder Layer Decoder Layer Decoder Layer I LM Head like Class Head LM Head Class Head LM Head Class Head LM Head Class Head LM Head Class Head LM Head sports , but dislike the being dogs sports you …. 0.2 0.1 0.01 0.25 0.22 …. 0.99 0.99 0.95 0.05 0.05 …. ✅ ✅ ✅ ❌ ❌ Vocabulary Next token probabilities SoftMax Linear LM Head (edim) (|V|) Sigmoid Linear Class Head (edim) (|V|) LM Class Figure 1: DIRECTOR employs a language model head and a classifier head at every step during left-right gen- eration, predicting the next token by combining the two probabilities. The classifier head is trained to direct generation away from undesirable sequences for exam- ple contradictions or repetitions (next token: “sports”) or toxic statements (next token: “you”), which the lan- guage model head may otherwise predict as likely. the standard language modeling objective. Stan- dard training can make use of …

Content truncated to 500 words