TIGEr: Text-to-Image Grounding for Image Caption Evaluation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2141–2152, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2141 TIGEr: Text-to-Image Grounding for Image Caption Evaluation Ming Jiang1, Qiuyuan Huang3, Lei Zhang3, Xin Wang2 Pengchuan Zhang3, Zhe Gan3, Jana Diesner1, Jianfeng Gao3 1University of Illinois at Urbana-Champaign, 2University of California, Santa Barbara 3Microsoft Research, Redmond {mjiang17,jdiesner}@illinois.edu, xwang@cs.ucsb.edu {qihua,leizhang,penzhan,zhe.gan,jfgao}@microsoft.com Abstract This paper presents a new metric called TIGEr for the automatic evaluation of image caption- ing systems. Popular metrics, such as BLEU and CIDEr, are based solely on text match- ing between reference captions and machine- generated captions, potentially leading to bi- ased evaluations because references may not fully cover the image content and natural language is inherently ambiguous. Building upon a machine-learned text-image ground- ing model, TIGEr allows to evaluate caption quality not only based on how well a cap- tion represents image content, but also on how well machine-generated captions match human-generated captions. Our empirical tests show that TIGEr has a higher consistency with human judgments than alternative exist- ing metrics. We also comprehensively assess the metric’s effectiveness in caption evaluation by measuring the correlation between human judgments and metric scores. 1 Introduction Image captioning is a research topic at the nexus of natural language processing and computer vision, and has a wide range of practical applications (Ho- dosh et al., 2013; Fang et al., 2015), such as image retrieval and human-machine interaction. Given an image as input, the task of image captioning is to generate a text that describes the image content. Overall, prior studies of this task have been focus- ing on two aspects: 1) building large benchmark datasets (Lin et al., 2014; Hodosh et al., 2013), and 2) developing effective caption generation al- gorithms (Karpathy and Li, 2015; Bernardi et al., 2016; Gan et al., 2017). Remarkable contributions that have been made, but assessing the quality of the generated captions is still an insufficiently ad- dressed issue. Since numerous captions with vary- ing quality can be produced by machines, it is im- Figure 1: An example of caption evaluation challenge. Given two candidate captions: (a) correctly describes image content, but most words in this caption are not contained in references, while (b) overlaps with refer- ences (“is holding a bottle of beer”), but involves wrong information (“a woman with straight hair”). Prior met- rics based only on text-level comparisons fail to as- sess the quality of candidate captions, while our met- ric (TIGEr) can detect this inconsistency by matching a caption with image content. To explain the TIGEr re- sult, we show the grounding weights of two illustrative image regions with each candidate. The proper text in- formation matched with each region is highlighted by colors. portant to propose an automatic evaluation metric that is highly consistent with human judges, easy to implement and interpret. Prior evaluation metrics typically considered several aspects, including content relevance, in- formation correctness, grammaticality, …

Content truncated to 500 words