Learning to Imagine: Visually-Augmented Natural Language Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9468–9481 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning to Imagine: Visually-Augmented Natural Language Generation Tianyi Tang1,4, Yushuo Chen1, Yifan Du1, Junyi Li1,3, Wayne Xin Zhao1,4 � , and Ji-Rong Wen1,2,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3DIRO, Université de Montréal 4Beijing Key Laboratory of Big Data Management and Analysis Methods steventianyitang@outlook.com chenyushuo1999@foxmail.com lijunyi@ruc.edu.cn {yifandu1999,batmanfly}@gmail.com Abstract People often imagine relevant scenes to aid in the writing process. In this work, we aim to uti- lize visual information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained language mod- els (PLMs) Learn to Imagine for Visually- augmented natural language gEneration. First, we imagine the scene based on the text: we use a diffusion model to synthesize high-quality images conditioned on the input texts. Sec- ond, we use CLIP to determine whether the text can evoke the imagination in a posterior way. Finally, our imagination is dynamic, and we conduct synthesis for each sentence rather than generate only one image for an entire paragraph. Technically, we propose a novel plug-and-play fusion layer to obtain visually-augmented rep- resentations for each text. Our vision-text fusion layer is compatible with Transformer- based architecture. We have conducted exten- sive experiments on four generation tasks us- ing BART and T5, and the automatic results and human evaluation demonstrate the effec- tiveness of our proposed method. We will re- lease the code, model, and data at the link: https://github.com/RUCAIBox/LIVE. 1 Introduction Natural language generation (NLG) is a fundamen- tal technique for supporting a variety of down- stream applications (Li et al., 2022b; Zhao et al., 2023), e.g., text summarization, story genera- tion, and data-to-text generation. As the main- stream NLG approach, pre-trained language mod- els (PLMs) can produce human-like text under the guidance of input conditions. Despite their suc- cess, these models are pre-trained on the text-only corpora, and they cannot well capture visually- grounded semantics, e.g., visual commonsense (Il- harco et al., 2021), making it difficult to achieve desired results when visual knowledge is required. � Corresponding author To improve the generation capacity of PLMs, ex- isting work has widely explored various methods to incorporate visual knowledge into models, which can be roughly divided into two lines of research. The first line designs specific visually-enhanced training tasks such as continual pre-training on text-image data (Cho et al., 2021) or knowledge distillation with vision-language models (Dai et al., 2022). However, these methods usually perform well only on multimodal generation tasks (e.g., vi- sual question answering) but not text generation tasks, due to the semantic disparity across modal- ities (Tan and Bansal, 2020). As the second line, several studies retrieve or synthesize images related to the input and then fuse the image representations into PLMs (Wang et al., 2022b; Zhu et al., 2022). However, they simply treat the input as a whole (even for …

Content truncated to 500 words