PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2953 - 2967 July 10-15, 2022 ©2022 Association for Computational Linguistics PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding Antoine Chaffin1,2, Vincent Claveau1, Ewa Kijak1 1CNRS, IRISA, Univ. Rennes 1, Campus de Beaulieu, 35000 Rennes, France 2IMATAG, 13 Rue Dupont-des-Loges, 35000 Rennes, France {antoine.chaffin, vincent.claveau, ewa.kijak}@irisa.fr Abstract Large language models (LM) based on Trans- formers allow to generate plausible long texts. In this paper, we explore how this generation can be further controlled at decoding time to satisfy certain constraints (e.g. being non- toxic, conveying certain emotions, using a spe- cific writing style, etc.) without fine-tuning the LM. Precisely, we formalize constrained gen- eration as a tree exploration process guided by a discriminator that indicates how well the associated sequence respects the constraint. This approach, in addition to being easier and cheaper to train than fine-tuning the LM, al- lows to apply the constraint more finely and dy- namically. We propose several original meth- ods to search this generation tree, notably the Monte Carlo Tree Search (MCTS) which pro- vides theoretical guarantees on the search effi- ciency, but also simpler methods based on re- ranking a pool of diverse sequences using the discriminator scores. These methods are evalu- ated, with automatic and human-based metrics, on two types of constraints and languages: re- view polarity and emotion control in French and English. We show that discriminator- guided MCTS decoding achieves state-of-the- art results without having to tune the lan- guage model, in both tasks and languages. We also demonstrate that other proposed decod- ing methods based on re-ranking can be really effective when diversity among the generated propositions is encouraged. 1 Introduction Generative language models exist for a long time, but with advent of the transformer architec- ture (Vaswani et al., 2017) and increasing comput- ing capabilities, they are now able to generate well written and long texts. In particular, large mod- els, such as the well known GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), have been used successfully for various applications: assist- ing writers, summarizing, augmentating data for subsequent NLP tasks, generating fake news (Ku- mar et al., 2020; Papanikolaou and Pierleoni, 2020; Zellers et al., 2019). Yet, beside the prompt used to initiate the generation process, there are few op- tions to have control on the generation process. Be- ing able to add some constraints on the generated texts is useful for various situations. For example, it allows to create texts that follow a certain writ- ing style, convey a certain emotion or polarity or to ensure that a generated summary contains cor- rect information. More critically, it can be used to prevent the inherent toxicity of language models trained on the internet, or to not reproduce gender or race stereotypes. So far, most methods neces- sitate to fine-tune the LM, so that it specifically learns to model this constraint, i.e. the constraint is –hopefully– …

Content truncated to 500 words