Transparent Human Evaluation for Image Captioning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3464 - 3478 July 10-15, 2022 ©2022 Association for Computational Linguistics Transparent Human Evaluation for Image Captioning Jungo Kasai♡∗ Keisuke Sakaguchi♣ Lavinia Dunagan♡ Jacob Morrison♡♠ Ronan Le Bras♣ Yejin Choi♡♣ Noah A. Smith♡♣ ♡Paul G. Allen School of Computer Science & Engineering, University of Washington ♣Allen Institute for AI ♠Department of Linguistics, University of Washington {jkasai,laviniad,jacobm00,yejin,nasmith}@cs.washington.edu {keisukes,ronanlb}@allenai.org Abstract We establish THUMB, a rubric-based human evaluation protocol for image captioning mod- els. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and in- clusive language). Our evaluations demon- strate several critical problems of the current evaluation practice. Human-generated cap- tions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric- based results reveal that CLIPScore, a recent metric that uses image features, better corre- lates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.1 1 Introduction Recent progress in large-scale training has pushed the state of the art in vision-language tasks (Li et al., 2020; Zhang et al., 2021, inter alia). One of these tasks is image captioning, whose objective is to generate a caption that describes the given image. The performance in image captioning has been pri- marily measured in automatic metrics (e.g., CIDEr, Vedantam et al., 2015; SPICE, Anderson et al., 2016) on popular benchmarks, such as MSCOCO (Lin et al., 2014) and Flickr8k (Hodosh et al., 2013). Use of these metrics is justified based on their corre- lation with human judgments collected in previous work (Hodosh et al., 2013; Elliott and Keller, 2014; Kilickaya et al., 2017, inter alia). ∗Work was done during an internship at AI2. 1All data are available at https://github.com/ jungokasai/THumB. Machines P R CIDEr A red fire hydrant spewing water on a street. 5 3 139.2 A red fire hydrant spraying water on a street. 5 3 205.2 Human A busted red fire hydrant spewing water all over a street creating a rainbow. 5 5 120.5 Figure 1: These machine captions are precise (in the scale of 1–5) but lose points in recall (i.e., coverage of salient information); they both ignore the rainbow in the picture. Automatic metrics, such as CIDEr, do not capture this failure. Continuous use of these previous human judg- ments, however, raises significant concerns for de- velopment of both captioning models and auto- matic metrics because of their lack of transparency. In previous work, annotators (crowdworkers, typi- cally) rate image captions directly (Hodosh et al., 2013), pairwise (Vedantam et al., 2015), or along multiple dimensions such …

Content truncated to 500 words