Improving Image Captioning Evaluation by Considering Inter References Variance

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985–994 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 985 Improving Image Captioning Evaluation by Considering Inter References Variance Yanzhi Yi and Hangyu Deng and Jinglu Hu Graduate School of Information, Production and Systems, Waseda University, 2-7 Hibikino, Wakamatsu, Kitakyushu-shi, Fukuoka, Japan, 808-0135 yiyanzhi@akane.waseda.jp, deng.hangyu@fuji.waseda.jp, jinglu@waseda.jp Abstract Evaluating image captions is very challenging partially due to the fact that there are mul- tiple correct captions for every single image. Most of the existing one-to-one metrics oper- ate by penalizing mismatches between refer- ence and generative caption without consider- ing the intrinsic variance between ground truth captions. It usually leads to over-penalization and thus a bad correlation to human judg- ment. Recently, the latest one-to-one metric BERTScore can achieve high human correla- tion in system-level tasks while some issues can be fixed for better performance. In this paper, we propose a novel metric based on BERTScore that could handle such a challenge and extend BERTScore with a few new fea- tures appropriately for image captioning eval- uation. The experimental results show that our metric achieves state-of-the-art human judg- ment correlation. 1 Introduction Image captioning is one of the key visual-linguistic tasks that asks for generated captions with specific images. Researchers look forward to inexpensive evaluation metrics that closely resemble human judgment, which remains a challenging task since most of the metrics can hardly get close to human judgment. Image captioning is a one-to-many task since each image can correspond to many possible cap- tions. Different captions may focus on different parts of the image; this not only creates a challenge for generating the captions (Dai et al., 2017; Venu- gopalan et al., 2017), but also for evaluating them. Most of the existing one-to-one evaluation metrics, however, overlook such a challenge. These one- to-one metrics (Lin, 2004; Vedantam et al., 2015; Zhang et al., 2019) ignore other reference captions since the score is computed by comparing the can- didate capture with one single reference caption. Figure 1: Intrinsic variance exists in a set of ground truth captions for an image. Differences between two references are commonly caused by two reasons: dif- ferent concerns or different descriptions. Different con- cerns mean different expressions between references are caused by different regions of interest in an image, while different descriptions mean references focus on the same part but use different ways to explain it. One- to-one metrics can hardly deal with the cases caused by different concerns. For example, they may regard Cand as a good caption compared with Ref1; while regard Cand as a bad caption compared with Ref2 or Ref3. When there are multiple reference captions, prior works compute individual scores for each reference caption and pool these scores together afterward. Intrinsic variance exists in a set of ground truth captions for an image, since different captions may have different concerns or descriptions. It’s chal- lenging to find a remedy for such over-penalization if the metric looks at …

Content truncated to 500 words