Visually-augmented pretrained language models for NLP tasks without images

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14912–14929 July 9-14, 2023 ©2023 Association for Computational Linguistics Visually-augmented Pretrained Language Models for NLP Tasks without Images Hangyu Guo1∗, Kun Zhou3,4∗, Wayne Xin Zhao2,4†, Qinyu Zhang1 and Ji-Rong Wen2,3,4 1School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen). 2Gaoling School of Artificial Intelligence, Renmin University of China. 3School of Information, Renmin University of China. 4Beijing Key Laboratory of Big Data Management and Analysis Methods. hyguo0220@gmail.com, francis_kun_zhou@163.com batmanfly@gmail.com, zqy@hit.edu.cn, jrwen@ruc.edu.cn Abstract Although pre-trained language models (PLMs) have shown impressive performance by text- only self-supervised training, they are found lack of visual semantics or commonsense. Ex- isting solutions often rely on explicit images for visual knowledge augmentation (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it is actually needed in specific inputs or tasks. To address these issues, we propose a novel Visually-Augmented fine-tuning approach that can be generally applied to various PLMs or NLP tasks, Without using any retrieved or gen- erated Images, namely VAWI. Experimental results show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and out- perform several competitive baselines on ten tasks. Our codes and data are publicly available at https://github.com/RUCAIBox/VAWI. 1 Introduction Recent years have witnessed the success of pre- trained language models (PLMs) (Qiu et al., 2020; Zhao et al., 2023), such as GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020), in a variety of natural language process (NLP) tasks. Since these PLMs are mostly trained on text-only corpus via self-supervised pre-training, they have been shown lack of visual commonsense (Liu et al., 2022) and real-world knowledge (Zhang et al., 2022). As a result, PLMs can’t well solve visually related lan- guage tasks 1, e.g., answering the color and size of common things, especially those requiring complex commonsense knowledge. To alleviate this problem, existing works mainly enhance PLMs by infusing visual information. Typ- ∗ Equal contributions. † Corresponding authors. 1In this work, we mainly focus on text-only NLP tasks that may benefit from external visual information, rather than visual-language tasks involving images. ically, given a text input, these studies firstly aug- ment the visual information from retrieved or gen- erated images about the input and then leverage their visual representations to improve PLMs on NLP tasks. Such an approach leads to visually- augmented pre-trained language models (VaLMs), where they adopt either visually-augmented pre- training (Tan and Bansal, 2020; Wang et al., 2022) or visually-augmented fine-tuning (Lu et al., 2022). Despite the effectiveness, there are two major short- comings in these methods. First, these methods of- ten rely on pre-learned complementary retrievers or generators, and also require time-consuming infer- ence to retrieve or generate proper images that are paired with the input. The above costly conditions largely limit the applicability of these approaches. Second, the retrieved or generated images are in- evitable to involve …

Content truncated to 500 words