Re-evaluating Automatic Metrics for Image Captioning

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 199–209, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Re-evaluating Automatic Metrics for Image Captioning Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem Hacettepe University Computer Vision Lab Dept. of Computer Engineering, Hacettepe University, Ankara, TURKEY kilickayamert@gmail.com,{aykut,nazli,erkut}@cs.hacettepe.edu.tr Abstract The task of generating natural language descriptions from images has received a lot of attention in recent years. Conse- quently, it is becoming increasingly im- portant to evaluate such image captioning approaches in an automatic manner. In this paper, we provide an in-depth evalua- tion of the existing image captioning met- rics through a series of carefully designed experiments. Moreover, we explore the utilization of the recently proposed Word Mover’s Distance (WMD) document met- ric for the purpose of image captioning. Our findings outline the differences and/or similarities between metrics and their rela- tive robustness by means of extensive cor- relation, accuracy and distraction based evaluations. Our results also demonstrate that WMD provides strong advantages over other metrics. 1 Introduction There has been a growing interest in research on integrating vision and language in natural lan- guage processing and computer vision communi- ties. As one of the key problems in this emerg- ing area, image captioning aims at generating natural descriptions of a given image (Bernardi et al., 2016). This is a challenging problem since it requires the ability to not only under- stand the visual content, but also to generate a linguistic description of that content. In this regard, it can be framed as a machine trans- lation task where the source language denotes the visual domain and the target language is a specific language such as English. The re- cently proposed deep image captioning studies follow this interpretation and model the process via an encoder-decoder architecture (Vinyals et al., 2015; Xu et al., 2015; Karpathy and Fei- Fei, 2015; Jia et al., 2015). These approaches have attained considerable success in the recent benchmarks such as FLICKR8K (Hodosh et al., 2013), FLICKR30K (Young et al., 2014) and MS COCO (Lin et al., 2014) as compared to the ear- lier techniques which explicitly detect objects and generate descriptions by using surface realization techniques (Kulkarni et al., 2013; Li et al., 2011; Elliott and Keller, 2013). With the size of the benchmark datasets be- coming larger and larger, evaluating image cap- tioning models has become increasingly impor- tant. Human-based evaluations become obsolete as they are costly to acquire and, more impor- tantly, not repeatable. Automatic evaluation met- rics are employed as an alternative to human eval- uation in both developing new models and com- paring them against the state-of-the-art. These metrics compute a score that indicates the simi- larity/dissimilarity between an automatically gen- erated caption and a number of human-written ref- erence (gold standard) descriptions. Some of these automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and TER (Snover et al., …

Content truncated to 500 words