CLIPScore: A Reference-free Evaluation Metric for Image Captioning

CLIPScore: A Reference-free Evaluation Metric for Image Captioning Jack Hessel† Ari Holtzman‡ Maxwell Forbes‡ Ronan Le Bras† Yejin Choi†‡ †Allen Institute for AI ‡Paul G. Allen School of Computer Science & Engineering, University of Washington {jackh,ronanlb}@allenai.org {ahai,mbforbes,yejin}@cs.washington.edu Abstract Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against cap- tions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empir- ical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M im- age+caption pairs from the web, can be used for robust automatic evaluation of image cap- tioning without the need for references. Ex- periments spanning several corpora demon- strate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming exist- ing reference-based metrics like CIDEr and SPICE. Information gain experiments demon- strate that CLIPScore, with its tight focus on image–text compatibility, is complemen- tary to existing reference-based metrics that emphasize text–text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based met- rics, e.g., news captions that require richer con- textual knowledge. 1 Introduction For most text generation tasks, reference-based n- gram overlap methods are still the dominant means of automatic evaluation. For image caption genera- tion, recent reference-based metrics have sought to transcend overlap by considering richer models of reference-candidate similarity: e.g., approximate scene graphs (Anderson et al., 2016), allowing reference-based methods to incorporate the image (Jiang et al., 2019; Lee et al., 2020). But, refer- ences can be expensive to collect and comparing reference captions - Two dogs are running towards each other across the sand. - Two dogs are running towards each other on a beach. - Two dogs run toward each other. clip clip sim. cos candidate Two dogs run towards each other on a marshy area. 0.83 clipScore vs traditional image captioning metrics clipScore How works clipScore ✓ clipScore humans 4/4 ✓74.0 vs ❌ bleu-1 33.3 ❌ 0.0 spice ❌ meteor 3.3 ❌ 0.5 cider-d vs clipScore humans 1/4 37.4 vs candidate A grey dog walks on top of a fallen tree in the woods. bleu-1 46.2 19.4 spice meteor 18.1 10.3 cider-d vs ❌ ❌ ✓ ✓ ✓ ✓ references - A helmeted boy flies through the air on a snowboard. - A snowboarder balancing on a wall. - A snowboarder in green grinds along the edge of a rail at night. - A snowboarder wearing a green jacket jumping a green railing. candidate Person snowboarding at a ski slope. references - A dirt biker in the forest. - A dirt biker rides his motorcycle through the woods. - A motocross bike is being ridden along a woodland path. - A person rides a motorbike on a dirt path surrounded by trees. Figure 1: …

Content truncated to 500 words