Measuring the Diversity of Automatic Image Descriptions

Proceedings of the 27th International Conference on Computational Linguistics, pages 1730–1741 Santa Fe, New Mexico, USA, August 20-26, 2018. 1730 Measuring the Diversity of Automatic Image Descriptions Emiel van Miltenburg Vrije Universiteit Amsterdam emiel.van.miltenburg@vu.nl Desmond Elliott University of Edinburgh d.elliott@ed.ac.uk Piek Vossen Vrije Universiteit Amsterdam piek.vossen@vu.nl Abstract Automatic image description systems typically produce generic sentences that only make use of a small subset of the vocabulary available to them. In this paper, we consider the production of generic descriptions as a lack of diversity in the output, which we quantify using established metrics and two new metrics that frame image description as a word recall task. This framing allows us to evaluate system performance on the head of the vocabulary, as well as on the long tail, where system performance degrades. We use these metrics to examine the diversity of the sentences generated by nine state-of-the-art systems on the MS COCO data set. We find that the systems trained with maximum likelihood objectives produce less diverse output than those trained with additional adversarial objectives. However, the adversarially-trained models only produce more types from the head of the vocabulary and not the tail. Besides vocabulary-based methods, we also look at the compositional capacity of the systems, specifically their ability to create compound nouns and prepositional phrases of different lengths. We conclude that there is still much room for improvement, and offer a toolkit to measure progress towards the goal of generating more diverse image descriptions. 1 Introduction Automatic image description is a challenging task because natural language and the visual world both have an unbounded range of variation (Bernardi et al., 2016). Computational image description models are trained to generalize over datasets of images with multiple human descriptions, however, much of the variation present in these descriptions is lost in a trained model. Dai et al. (2017) note that the descriptions generated by recurrent neural networks using a maximum-likelihood objective are “overly rigid and lacking in variability.” This rigidity and lack of variability in the output of state-of-the-art models is unfortunate because human descriptions are the exact opposite of this: Devlin et al. (2015) found that humans typically produce unique descriptions, i.e. only 4.8% of the human-described evaluation data in the MS COCO data set (Lin et al., 2014) also occur in the training data. The lack of variability in machine-generated text is not limited to automatic image description; it is a general problem in natural language generation. Simply put: automatically generated text quickly becomes boring or repetitive. Recent efforts to address this problem include using maximum mutual information as an objective function, rather than the likelihood of the output, to improve the variablity of a neural conversation model (Li et al., 2016). Castro Ferreira et al. (2016) focused on the deterministic nature of NLG systems, in the sense that they repeatedly use the same referential forms to refer to the same entity in longer stretches of text. They addressed this problem by explicitly training their model to mimic human variability …

Content truncated to 500 words