Belief Revision Based Caption Re-ranker with Visual Semantic Information

Proceedings of the 29th International Conference on Computational Linguistics, pages 5488–5506 October 12–17, 2022. 5488 Belief Revision based Caption Re-ranker with Visual Semantic Information Ahmed Sabir1, Francesc Moreno-Noguer2, Pranava Madhyastha3, Lluís Padró1 1 Computer Science Department, Universitat Politècnica de Catalunya, Barcelona, Spain 2 Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain 3 City, University of London, London, UK Abstract In this work, we focus on improving the cap- tions generated by image-caption generation systems. We propose a novel re-ranking ap- proach that leverages visual-semantic measures to identify the ideal caption that maximally cap- tures the visual information in the image. Our re-ranker utilizes the Belief Revision frame- work (Blok et al., 2003) to calibrate the orig- inal likelihood of the top-n captions by ex- plicitly exploiting the semantic relatedness be- tween the depicted caption and the visual con- text. Our experiments demonstrate the utility of our approach, where we observe that our re- ranker can enhance the performance of a typical image-captioning system without the necessity of any additional training or fine-tuning.1 1 Introduction Image caption generation is a task that predomi- nantly lies at the intersection of the areas of com- puter vision and natural language processing. The task is primarily aimed at generating a natural lan- guage description for a given image. Caption gener- ation systems usually consist of an image encoder that encodes a given image (usually by using a CNN) whose encoding is fed to a decoder (usu- ally by using a generative model such as RNN) to generate a natural language sentence which de- scribes the image succinctly. The most widely used approaches include a CNN-RNN end-to-end sys- tem (Vinyals et al., 2015; Anderson et al., 2018), end-to-end systems with attention that attend to specific regions of the image for generation (Xu et al., 2015; You et al., 2016) and systems with re- inforcement learning based methods (Rennie et al., 2017; Ren et al., 2017). Furthermore, recent ad- vances have resulted in end-to-end systems that use Transformer based architecture for language gener- ation and have become the current state-of-the-art 1https://github.com/ahmedssabir/ Belief-Revision-Score IM GENET Visual: orange, lemon, apple Visual B: a display of apples and oranges at a market Greedy: a fruit market with apple and orange B5: a fruit stand with apples and oranges re-ranking Visual hypothesis revision visual relatedness hypothesis init Language model Beam search Semantic similarity Caption baseline Figure 1: An overview of our hypothesis revision based visual re-ranker. We use the visual context from the image to revise and re-rank the most closely related caption to its visual context. These semantic relatedness measures are learned at the word-to-sentence level. In this example, we showcase our visual re-ranker (Visual Beam), a post-processing approach, which is able to re-rank the most ‘descriptive caption’ from the 5-Best Beam (Cornia et al., 2020). (Herdade et al., 2019; Huang et al., 2019; Cornia et al., 2020; Zhang et al., 2021b). While the state-of-the-art models generate cap- tions that are comparable to human level captions, they are …

Content truncated to 500 words