NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural Constraints

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9496–9510 July 9-14, 2023 ©2023 Association for Computational Linguistics NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural Constraints Mohaddeseh Bastan Stony Brook University mbastan@cs.stonybrook.edu Mihai Surdeanu University of Arizona msurdeanu@email.arizona.edu Niranjan Balasubramanian Stony Brook University niranjan@cs.stonybrook.edu Abstract Text generation often involves producing texts that also satisfy a given set of semantic con- straints. While most approaches for conditional text generation have primarily focused on lex- ical constraints, they often struggle to effec- tively incorporate syntactic constraints, which provide a richer language for approximating semantic constraints. We address this gap by introducing NEUROSTRUCTURAL DECODING, a new decoding algorithm that incorporates syn- tactic constraints to further improve the quality of the generated text. We build NEUROSTRUC- TURAL DECODING on the NeuroLogic Decod- ing (Lu et al., 2021b) algorithm, which en- ables language generation models to produce fluent text while satisfying complex lexical con- straints. Our algorithm is powerful and scal- able. It tracks lexico-syntactic constraints (e.g., we need to observe dog as subject and ball as object) during decoding by parsing the partial generations at each step. To this end, we adapt a dependency parser to generate parses for in- complete sentences. Our approach is evaluated on three different language generation tasks, and the results show improved performance in both lexical and syntactic metrics compared to previous methods. The results suggest this is a promising solution for integrating fine-grained controllable text generation into the conven- tional beam search decoding1. 1 Introduction Controllable text generation uses decoding algo- rithms that support lexico-semantic constraints to control what is included in the output. Check- ing for the presence or absence of certain words has been shown to improve applications such as recipe generation (Lu et al., 2021b), sentiment anal- ysis (Howard et al., 2022), and predicting implicit consequences of beliefs (Alexeeva et al., 2022). 1Code and data is available at https://stonybrooknlp. github.io/NeuroStructuralDecoding John introduced Kayla to Sarah Target Kayla introduced Sarah to John John introduced Kayla to Sarah nsubj obj Neurologic Decoding w/ lexical constraints {Kayla & introduce & Sarah & John} NeuroStructural Decoding w/ binary constraints {John, nsubj , introduce}& {Kayla, obj , introduce} Different Outputs obl Figure 1: An example that compares the output pro- duced by Neurologic Decoding with lexical constraints alone vs. the output generated by NEUROSTRUCTURAL DECODING with lexico-syntactic constraints. Most of the current work in constrained genera- tion focuses on handling lexical constraints (Miao et al., 2019; Lu et al., 2021b; Guo and Roth, 2021; Turcan et al., 2022). However, lexical constraints alone cannot directly support more complex se- mantic constraints such as the presence/absence of relations or events between multiple concepts. Consider the simple example in Figure 1, in which target sentence requires that different people have specific roles in an event. Lexical constraints can only check presence or absence of words (ala bag- of-words) and thus clearly cannot address such role requirements. This work targets methods that can handle these requirements. …

Content truncated to 500 words