Cross-validating Image Description Datasets and Evaluation Metrics

Cross-validating Image Description Datasets and Evaluation Metrics Josiah Wang and Robert Gaizauskas Department of Computer Science University of Sheffield, UK {j.k.wang, r.gaizauskas}@sheffield.ac.uk Abstract The task of automatically generating sentential descriptions of image content has become increasingly popular in recent years, resulting in the development of large-scale image description datasets and the proposal of various metrics for evaluating image description generation systems. However, not much work has been done to analyse and understand both datasets and the metrics. In this paper, we propose using a leave-one-out cross validation (LOOCV) process as a means to analyse multiply annotated, human-authored image description datasets and the various evaluation metrics, i.e. evaluating one image description against other human-authored descriptions of the same image. Such an evaluation process affords various insights into the image description datasets and evaluation metrics, such as the variations of image descriptions within and across datasets and also what the metrics capture. We compute and analyse (i) human upper-bound performance; (ii) ranked correlation between metric pairs across datasets; (iii) lower-bound performance by comparing a set of descriptions describing one image to another sentence not describing that image. Interesting observations are made about the evaluation metrics and image description datasets, and we conclude that such cross-validation methods are extremely useful for assessing and gaining insights into image description datasets and evaluation metrics for image descriptions. Keywords: Image Description, Evaluation Metrics, Vision and Language 1. Introduction The ability to describe the content of an image is essen- tial for various tasks such as image indexing and retrieval, and organising or browsing large image collections. With advances in visual object category recognition (e.g. being able to recognise and localise a car in an image), recent years have seen an upsurge of interest in moving beyond annotating images with isolated keywords to automatically generating sentence-level, natural language descriptions of image content (Three boys playing with a ball in the park). To facilitate progress and benchmarking on such automatic image description generation tasks, various image datasets with textual descriptions have been developed and various metrics proposed to evaluate such systems. The textual de- scriptions in such datasets are distinct from generic image captions (e.g. on Flickr or the Web, or in news articles) in that they aim to describe the literal visual content (actors, attributes, activities, scenes, etc.) of the corresponding im- age (A black dog jumping to catch a green ball in the field), and exclude non-literal, subjective opinions or semantic in- formation that requires external knowledge that cannot be determined from viewing the image alone (My adorable two-year-old poodle named Bobby loves playing with his favourite toy in the sun, pictured here in Hyde Park in Lon- don or Happiness is a day out with your pet). The image datasets provide textual descriptions written by multiple human annotators per image, and are often used during evaluation as gold-standard reference descriptions against a system generated candidate description. How- ever, little work has been done to analyse or evaluate the gold-standard descriptions against themselves, i.e. evalu- ating …

Content truncated to 500 words