Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2750 - 2762 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer Woojeong Jin1∗, Dong-Ho Lee1∗, Chenguang Zhu2, Jay Pujara1, Xiang Ren1 1Department of Computer Science, University of Southern California 2Microsoft Cognitive Services Research Group {woojeong.jin,dongho.lee,jpujara,xiangren}@usc.edu {chezhu}@microsoft.com Abstract Pre-trained language models are still far from human performance in tasks that need under- standing of properties (e.g. appearance, measur- able quantity) and affordances of everyday ob- jects in the real world since the text lacks such information due to reporting bias. In this work, we study whether integrating visual knowledge into a language model can fill the gap. We in- vestigate two types of knowledge transfer: (1) text knowledge transfer using image captions that may contain enriched visual knowledge and (2) cross-modal knowledge transfer using both images and captions with vision-language training objectives. On 5 downstream tasks that may need visual knowledge to solve the problem, we perform extensive empirical com- parisons over the presented objectives. Our ex- periments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings. 1 1 Introduction Pre-trained language models (PTLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and T5 (Raffel et al., 2020) have shown impressive results in various conventional natural language understanding (NLU) tasks by capturing syntactic and semantic knowledge from the pre- training tasks of masked language modeling and masked span infilling tasks on massive text corpora. Though yielding good performance on various NLU downstream tasks, these pre-training objec- tives suffer from a lack of out-of-domain knowl- edge that is not explicitly present in the pre-training corpus (Gururangan et al., 2020a; Petroni et al., 2021; Schick and Schütze, 2020). Specifically, one type of knowledge that models often struggle with is the visual knowledge of common objects such as attributes (e.g. appearance, measurable quantity) ∗Authors contributed equally. 1https://github.com/INK-USC/CMKT Interesting facts about orange ! 1. Orange elevates mood levels. 2. Orange are often grown in the Mediterranean. 3. Oranges facing the sunnier tend to be sweeter. Human Typical facts about orange … 1. Orange is a shape of circle. 2. Orange is a color of orange. Report Already knows... May not report Figure 1: Reporting Bias. People tend to report what interests them rather than typical and general facts. and affordances. This is because this kind of knowl- edge is rarely explicitly described in the training text due to reporting bias. For example, as shown in Figure 1, people tend to report what interests them rather than general facts such as a shape or color of oranges they already know. Towards better knowledge-enhanced PTLMs, re- cent works incorporate external knowledge bases (e.g., knowledge graph, dictionary) to inject entity knowledge into PTLMs (Zhang et al., 2019; Peters et al., 2019; Wang et al., 2021; Yu et al., 2021) or retrieve knowledge from external knowledge …

Content truncated to 500 words