Neural Caption Generation for News Images

Neural Caption Generation for News Images Vishwash Batra, Yulan He, George Vogiatzis School of Engineering and Applied Science, Aston University {batrav1, y.he9, g.vogiatzis}@aston.ac.uk Abstract Automatic caption generation of images has gained significant interest. It gives rise to a lot of interesting image-related applications. For example, it could help in image/video retrieval and management of vast amount of multimedia data available on the Internet. It could also help in development of tools that can aid visually impaired individuals in accessing multimedia content. In this paper, we particularly focus on news images and propose a methodology for automatically generating captions for news paper articles consisting of a text paragraph and an image. We propose several deep neural network architectures built upon Recurrent Neural Networks. Results on a BBC News dataset show that our proposed approach outperforms a traditional method based on Latent Dirichlet Allocation using both automatic evaluation based on BLEU scores and human evaluation. Keywords: Recurrent Neural Networks, Image caption generation, Deep learning, Order Embedding. 1. Introduction There is rich information available on the Internet. Many on-line news sites like CNN, Yahoo, BBC etc. publish im- ages with their stories and even provide photo feeds related to current events. These news sites are a good resource for multimedia files containing information in the form of videos, images and natural language texts. Presence of this vast amount of multimedia data has provided strong im- petus to develop machine-learning based applications that jointly model data from different modalities. For exam- ple, Ngiam et al. (2011) develops a speech recognition sys- tem where they jointly model audio and visual modality. They focus on learning representations of audio data which are coupled with the videos of the lips. Another such ap- plication is image caption or visual description generation, which aims to generate text descriptions for an image, often times capturing all the different objects depicted and their spatial relationships. News image caption generation, however, is different from the typical image captioning task. The input to news image caption generation is both a news article and its accompa- nying image, as opposed to the traditional image caption- ing task where the input is only an image. Hence, rather than enumerating objects in a given image and describing their properties or relationships to each other as in the tra- ditional image captioning task, the output of news image caption generation is informative text not only describing the key semantics conveyed in the given image, but also summarising the content of its relating news article (Berg et al., 2004). An example is shown in Figure 1. It can be seen that the captions of news images provide more infor- mation than what have been depicted in images only. For example, a reasonable caption for the second image would be “A building”. But its actual caption conveys much more information and it is evident that the text content of news articles would also need to be considered when generating good captions for news images. News caption generation tools …

Content truncated to 500 words