Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2136–2148 May 2-6, 2023 ©2023 Association for Computational Linguistics Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training Wenliang Dai† Zihan Liu∗⋄ Ziwei Ji† Dan Su† Pascale Fung† †The Hong Kong University of Science and Technology ⋄NVIDIA wenliang.dai@connect.ust.hk, pascale@ece.ust.hk Abstract Large-scale vision-language pre-trained (VLP) models are prone to hallucinate non-existent visual objects when generating text based on visual information. In this paper, we system- atically study the object hallucination prob- lem from three aspects. First, we examine recent state-of-the-art VLP models, showing that they still hallucinate frequently, and mod- els achieving better scores on standard metrics (e.g., CIDEr) could be more unfaithful. Second, we investigate how different types of image en- coding in VLP influence hallucination, includ- ing region-based, grid-based, and patch-based. Surprisingly, we find that patch-based features perform the best and smaller patch resolution yields a non-trivial reduction in object halluci- nation. Third, we decouple various VLP objec- tives and demonstrate that token-level image- text alignment and controlled generation are crucial to reducing hallucination. Based on that, we propose a simple yet effective VLP loss named ObjMLM to further mitigate object hal- lucination. Results show that it reduces object hallucination by up to 17.4% when tested on two benchmarks (COCO Caption for in-domain and NoCaps for out-of-domain evaluation). 1 Introduction Thanks to the advancement of large pre-trained Language Models (LMs) and Vision-Language Pre-training (VLP) methods, models are able to achieve surprisingly good performance in vision- conditioned text generation, e.g., image captioning. However, large LMs are found to generate unfaith- ful or nonsensical texts given the source input (Ji et al., 2022), which is called hallucination. The hal- lucination problem is also inherited by VLP mod- els (Alayrac et al., 2022), as they are still LMs that can understand visual information. VLP models of- ten generate fluent and seem appropriate sentences if we only see the text, but wrong when taking the ∗ Work done during PhD at HKUST. visual input into consideration. One major type of hallucination in VLP is known as object hal- lucination (Rohrbach et al., 2018), where models generate texts containing non-existent or inaccurate objects from the input images. Object hallucination in VLP models essentially limits their performance and raises safety concerns for industrial applica- tions. For example, in biomedical image caption- ing (Pavlopoulos et al., 2019), object hallucination reduces the accuracy of diagnosis and may lead to severe consequences for the patient. Despite the limitations and risks caused by object hallucination, this problem has not been studied in contemporary VLP works yet. To narrow down the aforementioned research gap, in this paper, we systematically investigate four fundamental research questions about object hallucination: 1) how much do modern VLP mod- els hallucinate? 2) how do different forms of image encoding affect object hallucination? 3) what are the effects of common VLP objectives on object hallucination? and 4) how to mitigate object hallu- cination in VLP models? For our …

Content truncated to 500 words