A Survey of Current Datasets for Vision and Language Research

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 207–213, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Survey of Current Datasets for Vision and Language Research Francis Ferraro1∗, Nasrin Mostafazadeh2∗, Ting-Hao (Kenneth) Huang3, Lucy Vanderwende4, Jacob Devlin4, Michel Galley4, Margaret Mitchell4 Microsoft Research 1 Johns Hopkins University, 2 University of Rochester, 3 Carnegie Mellon University, 4 Corresponding authors: {lucyv,jdevlin,mgalley,memitc}@microsoft.com Abstract Integrating vision and language has long been a dream in work on artificial intel- ligence (AI). In the past two years, we have witnessed an explosion of work that brings together vision and language from images to videos and beyond. The avail- able corpora have played a crucial role in advancing this area of research. In this paper, we propose a set of quality met- rics for evaluating and analyzing the vi- sion & language datasets and categorize them accordingly. Our analyses show that the most recent datasets have been us- ing more complex language and more ab- stract concepts, however, there are differ- ent strengths and weaknesses in each. 1 Introduction Bringing together language and vision in one in- telligent system has long been an ambition in AI research, beginning with SHRDLU as one of the first vision-language integration systems (Wino- grad, 1972) and continuing with more recent at- tempts on conversational robots grounded in the visual world (Kollar et al., 2013; Cantrell et al., 2010; Matuszek et al., 2012; Kruijff et al., 2007; Roy et al., 2003). In the past few years, an influx of new, large vision & language corpora, along- side dramatic advances in vision research, has sparked renewed interest in connecting vision and language. Vision & language corpora now provide alignments between visual content that can be rec- ognized with Computer Vision (CV) algorithms and language that can be understood and generated using Natural Language Processing techniques. Fueled in part by the newly emerging data, re- search that blends techniques in vision and in lan- guage has increased at an incredible rate. In just ∗F.F. and N.M. contributed equally to this work. the past year, recent work has proposed meth- ods for image and video captioning (Fang et al., 2014; Donahue et al., 2014; Venugopalan et al., 2015), summarization (Kim et al., 2015), refer- ence (Kazemzadeh et al., 2014), and question an- swering (Antol et al., 2015; Gao et al., 2015), to name just a few. The newly crafted large-scale vi- sion & language datasets have played a crucial role in defining this research, serving as a foundation for training/testing and helping to set benchmarks for measuring system performance. Crowdsourcing and large image collections such as those provided by Flickr1 have made it possible for researchers to propose methods for vi- sion and language tasks alongside an accompany- ing dataset. However, as more and more datasets have emerged in this space, it has become un- clear how different methods generalize beyond the datasets they are evaluated on, and what data may be useful for moving the field …

Content truncated to 500 words