NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4288–4299 June 6–11, 2021. ©2021 Association for Computational Linguistics 4288 NEUROLOGIC DECODING: (Un)supervised Neural Text Generation with Predicate Logic Constraints Ximing Lu†‡ Peter West†‡ Rowan Zellers†‡ Ronan Le Bras‡ Chandra Bhagavatula‡ Yejin Choi†‡ †Paul G. Allen School of Computer Science & Engineering, University of Washington ‡Allen Institute for Artificial Intelligence {lux32, pawest, rowanz, yejin}@cs.washington.edu {ronanlb, chandrab}@allenai.org Abstract Conditional text generation often requires lex- ical constraints, i.e., which words should or shouldn’t be included in the output text. While the dominant recipe for conditional text gener- ation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to fol- low the underlying constraints reliably, even when supervised with large amounts of task- specific examples. We propose NEUROLOGIC DECODING, a sim- ple yet effective algorithm that enables neu- ral language models – supervised or not – to generate fluent text while satisfying complex lexical constraints. Our approach is power- ful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equiva- lent to conventional beam search. Empirical results on four benchmarks show that NEUROLOGIC DECODING outperforms previous approaches, including algorithms that handle a subset of our constraints. More- over, we find that unsupervised models with NEUROLOGIC DECODING often outperform supervised models with conventional decod- ing, even when the latter is based on consid- erably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms. 1 Introduction Text generation applications often need to incorpo- rate semantic constraints, i.e., what words should and shouldn’t appear in the output generation. Con- sider the task of generating a recipe from a set of ingredients (Kiddon et al., 2016), such as ‘garlic,’ ‘steak’, and ‘soy sauce’ (Figure 1). A generated recipe should cover all of those ingredients, without hallucinating new ones (such as ‘pork’ or ‘beans’). This restriction, like others in Figure 1 for other input Scenario food | table | sit | front The man sat with his food at the front of the table The food is in front of you sit at the table. a table of food sits in front of three people Concept-Set COMMONGEN (Lin et al., 2019) Constraints (food ⋁ foods) ⋀ (table ⋁ tables) ⋀ (sit ⋁ sits ⋁ sat ⋁ sitting) ⋀ (front ⋁ fronts) input output out Target out The physician told the baker that she had cancer. Der Arzt sagte dem Bäckerin, dass er Krebs habe. Evaluate Gender Bias in MT (Stanovsky et al., 2019) Bäckerin Bäcker Constraints (Ärztin ⋁ Arzt) ⋀ (Bäckerin ⋀ ¬ Bäcker) Source Recipe Generation (Kiddon et al., 2016) input output Dish name 2 tsp butter, 1 beef steak, 1/4 tsp soy sauce, 1 tsp parsley, 1/8 tsp salt , 1/2 tsp garlic garlic butter steak Ingredients Constraints butter ⋀ (beef ⋁ steak …

Content truncated to 500 words