InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3171–3185 July 9-14, 2023 ©2023 Association for Computational Linguistics InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation Anwen Hu1, Shizhe Chen2, LiangZhang1, Qin Jin1∗ 1School of Information, Renmin University of China 2INRIA {anwenhu,zhangliang00,qjin}@ruc.edu.cn shizhe.chen@inria.fr Abstract Automatic image captioning evaluation is criti- cal for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and infor- mative. Instead, we humans can easily iden- tify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than exist- ing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation. Our code and datasets are publicly available at https://github. com/HAWLYQ/InfoMetIC. 1 Introduction Image captioning aims to automatically gener- ate natural language sentences to describe image contents. Recently, there are significant break- throughs in image captioning such as attention- based model architectures (Anderson et al., 2018; Pan et al., 2020; Hu et al., 2020, 2021) and vision- and-language pretraining (VLP) (Zhou et al., 2020; Xia et al., 2021; Li et al., 2022b; Xu et al., 2021; Li et al., 2022a). However, as groundtruth image descriptions are extremely diverse and subjective, evaluating the image captioning performance re- mains a considerable challenge. The most widely used image captioning met- rics such as METEOR (Banerjee and Lavie, 2005), ∗*Corresponding Author. 1.References: A brown and white dog is running. 2. A dog runs on the snow with a package. 3. There is a hat buried in the snow. Candidate 1(C1): A dog on the snow. Candidate 1(C2): A dog with a hat is on the snow. Candidate 2(C3): A dog with a bag is on the snow. 0.27 0.31 0.78 6.33 0.37 0.40 0.76 6.05 0.37 0.27 0.84 8.14 ✔ ✘ ✔ ✔ ✔ ✔ 4.78 6.45 1.27 1.69 C1 C2 Text Precision Vision Recall Overall Ours: InfoMetIC (w/o ref) SPICECLIP-S METEOR (w/ ref)(w/o ref) (w/ ref) dog✔ hat✘ snow✔ dog✔ bag✔ snow✔ C3 5.71 dog✔ snow✔ ✔ ✘ ✔ 0.62 Figure 1: Comparison of existing metrics and our infor- mative metric (InfoMetIC). ‘w/ ref’ and ‘w/o ref’ mean using references or not. CIDEr (Vedantam et al., 2015a) and SPICE (Ander- son et al., 2016) utilize human-written descriptions of images as references and measure similarities between generated captions and references for eval- uation. Such reference-based approaches suffer from two major limitations. Firstly, these metrics mainly evaluate caption …

Content truncated to 500 words