Apples to Oranges: Evaluating Image Annotations from Natural Language Processing Systems

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 172–181, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Apples to Oranges: Evaluating Image Annotations from Natural Language Processing Systems Rebecca Mason and Eugene Charniak Brown Laboratory for Linguistic Information Processing (BLLIP) Brown University, Providence, RI 02912 {rebecca,ec}@cs.brown.edu Abstract We examine evaluation methods for systems that automatically annotate images using co- occurring text. We compare previous datasets for this task using a series of baseline mea- sures inspired by those used in information re- trieval, computer vision, and extractive sum- marization. Some of our baselines match or exceed the best published scores for those datasets. These results illuminate incorrect as- sumptions and improper practices regarding preprocessing, evaluation metrics, and the col- lection of gold image annotations. We con- clude with a list of recommended practices for future research combining language and vi- sion processing techniques. 1 Introduction Automatic image annotation is an important area with many applications such as tagging, generat- ing captions, and indexing and retrieval on the web. Given an input image, the goal is to generate rel- evant descriptive keywords that describe the visual content of the image. The Computer Vision (CV) literature contains countless approaches to this task, using a wide range of learning techniques and visual features to identify aspects such as objects, people, scenes, and events. Text processing is computationally less expensive than image processing and easily provides informa- tion that is difficult to learn visually. For this reason, most commerical image search websites identify the semantic content of images using co-occurring text exclusively. But co-occurring text is also a noisy source for candidate annotations, since not all of the text is visually relevant. Techniques from Natural Language Processing help align descriptive words and images. Some examples of previous research use named-entity recognition to identify people in images (Deschacht and Moens, 2007); term associa- tion to estimate the “visualness” of candidate anno- tations (Boiy et al., 2008; Leong et al., 2010); and topic models to annotate images given both visual and textual features (Feng and Lapata, 2010b). Image annotation using NLP is still an emerging area with many different tasks, datasets, and eval- uation methods, making it impossible to compare many recent systems to each other. Although there is some effort being made towards establishing shared tasks1, it is not yet clear which kinds of tasks and datasets will provide interesting research questions and practical applications in the long term. Until then, establishing general “best practices” for NLP image annotation will help advance and legitimitize this work. In this paper, we propose some good prac- tices and demonstrate why they are important. 2 Image Annotation Evaluation in CV and NLP In this section, we first review related work in im- age annotation evaluation in computer vision, spe- cific challenges, and proposed solutions. We then relate these challenges to the NLP image annotation task and some of the specific problems we propose to address. 1http://imageclef.org/ 172 …

Content truncated to 500 words