Discourse-Aware Soft Prompting for Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4570 - 4589 December 7-11, 2022 ©2022 Association for Computational Linguistics Discourse-Aware Soft Prompting for Text Generation Marjan Ghazvininejad Vladimir Karpukhin Vera Gor Asli Celikyilmaz Meta AI Abstract Current efficient fine-tuning methods (e.g., adapters (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency. While showing strong performance on some generation tasks, they don’t generalize across all generation tasks. We show that soft-prompt based conditional text generation can be im- proved with simple and efficient methods that simulate modeling the discourse structure of human written text. We investigate two design choices: First, we apply hierarchical blocking on the prefix parameters to simulate a higher- level discourse structure of human written text. Second, we apply attention sparsity on the pre- fix parameters at different layers of the net- work and learn sparse transformations on the softmax-function. We show that structured de- sign of prefix parameters yields more coherent, faithful and relevant generations than the base- line prefix-tuning on all generation tasks. 1 Introduction Recent advances in pre-trained langauge models (PLMs) (Lewis et al., 2020; Raffel et al., 2020; Rad- ford et al., 2019) have made great impact on text generation research, especially when they are fine- tuned on downstream tasks such as summarization, data-to-text generation, long-question answering, etc. Consequent research have shown that PLMs’ impact can further be improved when trained with more parameters, on more data and with more com- pute (GPT-3 (Brown et al., 2020), Megatron (Smith et al., 2022)). On the flip side, storing larger LMs or fully fine-tuning them (updating all the parame- ters) on downstream tasks usually causes resource or over-fitting issues. To mitigate fine-tuning issues, recent work have proposed prompt-based learning (Liu et al., 2021a), which focus on learning textual prompts to steer PLMs’ continuation towards desired output while keeping the model parameters frozen. While pro- viding strong control of the PLMs, such prompt en- gineering could be time consuming requiring man- ual crafting. There is a growing research direction under prompt learning towards lightweight fine- tuning (Houlsby et al., 2019; Lester et al., 2021), which update only a small number of existing or extra parameters while keeping most of the origi- nal pre-trained parameters frozen. Among them is prefix-tuning (Li and Liang, 2021), which prepends tunable continuous task-specific prompt vectors called prefixes to the input and only trains these continuous prompts during fine-tuning. Although prefix-tuning can yield comparable results to full fine-tuning on some generation tasks, it did not generalize to tasks like abstractive summarization. In this work, we focus on prefix-tuning and in- vestigate ways to improve its generalization on text generation tasks. We start asking the following questions that motivates our design choices: (1) Do different parts of the transformer network process the prefix parameters more efficiently?; (2) Do pre- fix parameters capture high-level discourse …

Content truncated to 500 words