Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5882–5895 October 12–17, 2022. 5882 Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation Fei-Tzin Lee∗, Miguel Ballesteros†, Feng Nan† and Kathleen McKeown∗ * Computer Science Department, Columbia University † Amazon AI Labs {feitzin,kathy}@cs.columbia.edu {ballemig,nanfen}@amazon.com Abstract Large pretrained language models offer pow- erful generation capabilities, but cannot be re- liably controlled at a sub-sentential level. We propose to make such fine-grained control pos- sible in pretrained LMs by generating text di- rectly from a semantic representation, Abstract Meaning Representation (AMR), which is aug- mented at the node level with syntactic control tags. We experiment with English-language generation of three modes of syntax relevant to the framing of a sentence - verb voice, verb tense, and realization of human entities - and demonstrate that they can be reliably controlled, even in settings that diverge drastically from the training distribution. These syntactic as- pects contribute to how information is framed in text, something that is important for appli- cations such as summarization which aim to highlight salient information. 1 Introduction Language models pretrained on enormous corpora have become a staple in natural language process- ing because of their power and adaptability (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020). These models exhibit strong performance across a range of applications, but are not inherently control- lable beyond the choice of input. To enforce spe- cific constraints on their output, we must introduce an additional mechanism for control, whether by inserting control codes during pretraining (Keskar et al., 2019) or by adding components to the model (Dathathri et al., 2020). These prior methods allow specification of high-level attributes (e.g., topic or sentiment) but leave the specifics of sub-sentential realization to the model. In contrast, we present a method for controlling generation of specific verbs and entities within a sentence. We investigate the setting in which a pretrained language model is used not as an end- to-end solution, but rather to directly generate text from a predefined content plan. Such a content plan may be created at an intermediate stage in a given task (e.g., as the product of content selection over a document graph in AMR-based summarization) but is not inherently task-specific. Specifically, we focus on the controllability of BART when it is fine-tuned to generate text from an intermediate Abstract Meaning Representation (AMR) (Banarescu et al., 2013), a form of graphi- cal semantic representation. We choose AMR be- cause it is relatively widely used, including as an intermediate representation in summarization (Liu et al., 2015; Hardy and Vlachos, 2018) and machine translation (Song et al., 2019), and the problem of AMR-to-text generation is well-studied (Konstas et al., 2017; Wang et al., 2020; Bai et al., 2020; Zhang et al., 2020). We focus on BART because of its competitive performance in summarization, a task where we feel our work is particularly appli- cable, as document-level content graphs have been shown to be useful intermediate representations in …

Content truncated to 500 words