Generating Natural Questions About an Image

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1802–1813, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Generating Natural Questions About an Image Nasrin Mostafazadeh1, Ishan Misra2, Jacob Devlin3, Margaret Mitchell3 Xiaodong He3, Lucy Vanderwende3 1 University of Rochester, 2 Carnegie Mellon University, 3 Microsoft Research nasrinm@cs.rochester.edu, lucyv@microsoft.com Abstract There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering ques- tions about images. These tasks have fo- cused on literal descriptions of the image. To move beyond the literal, we choose to explore how questions about an image are often directed at commonsense inference and the abstract events evoked by objects in the image. In this paper, we introduce the novel task of Visual Question Gener- ation (VQG), where the system is tasked with asking a natural and engaging ques- tion when shown an image. We provide three datasets which cover a variety of im- ages from object-centric to event-centric, with considerably more abstract training data than provided to state-of-the-art cap- tioning systems thus far. We train and test several generative and retrieval mod- els to tackle the task of VQG. Evaluation results show that while such models ask reasonable questions for a variety of im- ages, there is still a wide gap with human performance which motivates further work on connecting images with commonsense knowledge and pragmatics. Our proposed task offers a new challenge to the commu- nity which we hope furthers interest in ex- ploring deeper connections between vision & language. 1 Introduction We are witnessing a renewed interest in interdis- ciplinary AI research in vision & language, from descriptions of the visual input such as image cap- tioning (Chen et al., 2015; Fang et al., 2014; Don- ahue et al., 2014; Chen et al., 2015) and video Natural Questions: - Was anyone injured in the crash? - Is the motorcyclist alive? - What caused this accident? Generated Caption: - A man standing next to a motorcycle. Figure 1: Example image along with its natural questions and automatically generated caption. transcription (Rohrbach et al., 2012; Venugopalan et al., 2015), to testing computer understanding of an image through question answering (Antol et al., 2015; Malinowski and Fritz, 2014). The most established work in the vision & language com- munity is ‘image captioning’, where the task is to produce a literal description of the image. It has been shown (Devlin et al., 2015; Fang et al., 2014; Donahue et al., 2014) that a reasonable language modeling paired with deep visual features trained on large enough datasets promise a good perfor- mance on image captioning, making it a less chal- lenging task from language learning perspective. Furthermore, although this task has a great value for communities of people who are low-sighted or cannot see in all or some environments, for oth- ers, the description does not add anything to what a person has already perceived. The popularity of …

Content truncated to 500 words