Quality Estimation for Image Captions Based on Large-scale Human Evaluations

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3157–3166 June 6–11, 2021. ©2021 Association for Computational Linguistics 3157 Quality Estimation for Image Captions Based on Large-scale Human Evaluations Tomer Levinboim Ashish V. Thapliyal Piyush Sharma Radu Soricut Google Research Venice, CA 90291 {tomerl,asht,piyushsharma,rsoricut}@google.com Abstract Automatic image captioning has improved sig- nificantly over the last few years, but the prob- lem is far from being solved, with state of the art models still often producing low qual- ity captions when used in the wild. In this paper, we focus on the task of Quality Es- timation (QE) for image captions, which at- tempts to model the caption quality from a hu- man perspective and without access to ground- truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation pro- cess that collects coarse-grained caption anno- tations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the col- lected ratings and establish baseline models for this new QE task. Finally, we further col- lect fine-grained caption quality annotations from trained raters, and use them to demon- strate that QE models trained over the coarse ratings can effectively detect and filter out low- quality image captions, thereby improving the user experience from captioning systems. 1 Introduction Image captioning technology produces automatic image descriptions using natural language with the goal of being consumed by end-users that may not be able to directly access the images. This need arises either because the user has a permanent con- dition (accessibility for visually impaired people), or due to a temporary situation where the user can- not use the visual modality (such as limited band- width, or smart voice-assistant). In any of these situations, exposing the end-users to a generated caption that is incorrect negatively impacts user- trust, as it can have undesirable consequences for how they act next (for example, how they comment on a social-media site based on their misguided understanding). In this paper, we propose to mitigate such risks through Quality Estimation (QE) of image captions. That is, we propose to automatically compute a quality estimation score QE(image, caption) for a generated caption, and use it to control the quality of the captions presented to the user. For example, by filtering out captions with a low QE score (below a carefully chosen threshold), only high scoring captions would be served thereby minimizing the risks associated with low-quality captions. We emphasize two aspects of QE that have guided us in our design choices: First, the QE task is distinct from the model selection task: model selection measures output similarity to a fixed, ground-truth annotated dataset during training time (with traditional offline solutions such as CIDEr and SPICE). In contrast, a QE model estimates the caption quality with respect …

Content truncated to 500 words