VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6539–6550 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6539 VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions Pranava Madhyastha⇔, Josiah Wang⇔ and Lucia Specia Department of Computing Imperial College London, UK {pranava, josiah.wang, l.specia}@imperial.ac.uk Abstract We address the task of evaluating image de- scription generation systems. We propose a novel image-aware metric for this task: VIFIDEL. It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description. The metric is also able to take into account the rel- ative importance of objects mentioned in hu- man reference descriptions during evaluation. Even if these human reference descriptions are not available, VIFIDEL can still reliably eval- uate system descriptions. The metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on and rely exclusively on human references. 1 Introduction A popular task at the intersection of computer vision and natural language is image description generation (IDG), i.e. the task of generating as output a sentence describing the visual content of a given input image. While a variety of methods have been proposed for this task (Kulkarni et al., 2011; Li et al., 2011; Vinyals et al., 2015), its eval- uation is still an understudied problem. Evalua- tion of IDG is currently performed in two ways: (i) human judgment; (ii) automatic metrics. Hu- man judgments evaluate either the overall quality of descriptions or specific criteria in isolation (rel- evance, fluency, etc.). Such methods, however, can be subjective and expensive to scale. Automatic metrics address the scalability is- sue by comparing candidate descriptions against human-authored reference descriptions. These metrics conflate various criteria implicitly into a ⇔ Pranava Madhyastha and Josiah Wang contributed equally as joint first authors to this work. single evaluation assumption, i.e. a good descrip- tion is one that is similar to one or more human- authored descriptions, presuming that these gold descriptions are fluent, correct and relevant to the image. Existing automatic metrics are thus use- ful for measuring the quality of descriptions as a whole, but this makes it difficult for the specific capabilities of IDG systems to be inspected. We argue that a fine-grained metric measuring specific criteria would be more useful in under- standing how an IDG system is better than an- other. We focus on one such criterion, visual fi- delity1. This criterion aims to measure how faith- ful a description is with respect to what is depicted in the image (i.e. systems should be rewarded for describing elements depicted in the image and pe- nalised for describing things that are not depicted). For that, we propose to take image content into account when evaluating descriptions, in contrast to previous work (§2) that rely solely on words in the reference descriptions. Given that most datasets are crowd-sourced, …

Content truncated to 500 words