Image Caption Generation for News Articles

Proceedings of the 28th International Conference on Computational Linguistics, pages 1941–1951 Barcelona, Spain (Online), December 8-13, 2020 1941 Image Caption Generation for News Articles Zhishen Yang Tokyo Institute of Technology zhishen.yang @nlp.c.titech.ac.jp Naoaki Okazaki Tokyo Institute of Technology okazaki @c.titech.ac.jp Abstract In this paper, we address the task of news-image captioning, which generates a description of an image given the image and its article body as input. This task is more challenging than the conventional image captioning, because it requires a joint understanding of image and text. We present a Transformer model that integrates text and image modalities and attends to textual fea- tures from visual features in generating a caption. Experiments based on automatic evaluation metrics and human evaluation show that an article text provides primary information to repro- duce news-image captions written by journalists. The results also demonstrate that the proposed model outperforms the state-of-the-art model. In addition, we also confirm that visual features contribute to improving the quality of news-image captions. 1 Introduction Image captioning, i.e., automatic generation of a natural language description from an image, has re- ceived much attention from both fields of Computer Vision (CV) and Natural Language Processing (NLP) (Vinyals et al., 2015; Xu et al., 2015; Karpathy and Fei-Fei, 2015). Image captioning is important not only in practical application (e.g., automatic indexing of images) but also as a challenge of image understanding (e.g., recognition of objects, object-object relationships, and scenes). In this paper, we address a more advanced task, news-image captioning: this task generates a de- scription of an image, given the image and its article body as input. The news-image captioning task is different from the conventional image captioning, which receives only an image as input. In other words, news-image captioning requires a mutual understanding of image and text. Early work proposed a two-stage approach for news-image captioning (Feng and Lapata, 2013; Tariq and Foroosh, 2017). The first stage annotates keywords to a given image and text, and the second stage realizes a description based on the extracted keywords. Recent work presented an end-to-end approach that integrates image and text features in deep neural networks (Ramisa et al., 2018; Batra et al., 2018; Biten et al., 2019). However, the previous studies did not focus on the usefulness of text in the news-image captioning task, extending the conventional models for image captioning to incorporate text features. Figure 1 shows an example of a news-image caption. It may be difficult to recognize which is the central object in the image, for example, people, violins, and stick (bow). Also, the caption includes much information (e.g., Juilliard Orchestra, Vladimir Jurowsky, and Alice Tully Hall) that may be hard to tell only from the image. This kind of example is rather common in news articles, where a text is the major medium of information, and an image and its caption provide additional explanations that support the text. However, no previous work explored a method that integrated an article text seamlessly with an image in the task …

Content truncated to 500 words