REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1475–1480, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1475 REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning Ming Jiang1, Junjie Hu2, Qiuyuan Huang3, Lei Zhang3, Jana Diesner1, Jianfeng Gao3 1University of Illinois at Urbana-Champaign, 2Carnegie Mellon University 3Microsoft Research, Redmond {mjiang17,jdiesner}@illinois.edu, junjieh@cs.cmu.edu {qihua,leizhang,jfgao}@microsoft.com Abstract Popular metrics used for evaluating image cap- tioning systems, such as BLEU and CIDEr, provide a single score to gauge the system’s overall effectiveness. This score is often not informative enough to indicate what spe- cific errors are made by a given system. In this study, we present a fine-grained evalua- tion method REO for automatically measur- ing the performance of image captioning sys- tems. REO assesses the quality of captions from three perspectives: 1) Relevance to the ground truth, 2) Extraness of the content that is irrelevant to the ground truth, and 3) Omission of the elements in the images and human ref- erences. Experiments on three benchmark datasets demonstrate that our method achieves a higher consistency with human judgments and provides more intuitive evaluation results than alternative metrics.1 1 Introduction Image captioning is an interdisciplinary task that aims to automatically generate a text description for a given image. The task is fundamental to a wide range of applications, including image re- trieval (Rui et al., 1999) and vision language nav- igation (Wang et al., 2019). Though remarkable progress has been made (Gan et al., 2017; Karpa- thy and Li, 2015), the automatic evaluation of im- age captioning systems remains a challenge, par- ticularly with respect to quantifying the genera- tion errors made by these systems (Bernardi et al., 2016). Existing metrics for caption evaluation can be grouped into two categories: 1) rule-based met- rics (Papineni et al., 2002; Vedantam et al., 2015) that are based on exact string matching, and 2) learning-based metrics (Cui et al., 2018; Sharif 1Code is released at https://github.com/ SeleenaJM/CapEval. • A"man"instruc,ng"a"group"of"kids"on"a"soccer"field." • A"soccer"coach"is"instruc,ng"the"children"on"the" field." • Pair"of"adult"males"with"group"of"small"children"with" soccer"balls."" References( Candidate(A:"Two"men"are"teaching"kids" football"in#a#shopping#mall#with#many# stores." Candidate(B:"Two"men"are"teaching" football."(missing#“kids”)# BLEU4:"""0.00( CIDEr:"0.05" SPICE:"0.05( BLEU4:"""0.00( CIDEr:"0.00" SPICE:"0.06( REO_R:""0.28" REO_E:"9.39" REO_O:"10.64" REO_R:""0.26" REO_E:"10.40" REO_O:"9.32" Figure 1: An example of caption evaluation. Given two caption candidates, even though Caption A covers more image information than Caption B (e.g., missing ”kids”), Caption A contains extra irrelevant description (highlighted in red). Prior metrics (e.g., BLEU4) only provide an overall quality score, which is difficult to infer specific description mistakes in a caption. In con- trast, REO provides three indicators (i.e., relevance, ex- traness, and omission) that can properly achieve a fine- grained assessment for each caption. et al., 2018) that predict the probability of a test- ing caption as a human-generated caption by us- ing a learning model. In general, prior work has shown that description adequacy with respect to the ground truth data is a main concern for eval- uating text generation systems (Gatt and Krah- mer, 2018). Though this aspect has …

Content truncated to 500 words