Visual News: Benchmark and Challenges in News Image Captioning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6761–6771 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6761 Visual News: Benchmark and Challenges in News Image Captioning Fuxiao Liu University of Maryland fl3es@umd.edu Yinghan Wang∗ Amazon Alexa yinghanw@amazon.com Tianlu Wang University of Virginia tianlu@virginia.edu Vicente Ordonez Rice University vicenteor@rice.edu Abstract We propose Visual News Captioner, an entity- aware model for the task of news image cap- tioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author informa- tion, and other metadata. Unlike the standard image captioning task, news images depict sit- uations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and tex- tual features to generate captions with richer information such as events and entities. More specifically, built upon the Transformer archi- tecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms, which are designed to generate named entities more accurately. Our method utilizes much fewer parameters while achieving slightly better prediction re- sults than competing methods. Our larger and more diverse Visual News dataset further high- lights the remaining challenges in captioning news images. 1 Introduction Image captioning is a language and vision task that has received considerable attention and where important progress has been made in recent years (Vinyals et al., 2015; Fang et al., 2015; Xu et al., 2015; Lu et al., 2018b; Anderson et al., 2018). This field has been fueled by recent advances in both visual representation learning and text genera- tion, and also by the availability of image-text paral- lel corpora such as the Common Objects in Context (COCO) Captions dataset (Chen et al., 2015). While COCO contains enough images to train reasonably good captioning models, it was col- lected so that objects depicted in the images are biased toward a limited set of everyday objects. Moreover, while it provides high-quality human ∗Work completed before joining Amazon. President Obama and Mitt Romney debate in Hempstead NY on Tuesday. Virginia Cavaliers fans celebrate on the court after the Cavaliers game against the Duke Blue Devils at John Paul Jones Arena. A baseball player hitting the ball during the game. A bunch of people who are holding red umbrellas. Figure 1: Examples from our Visual News dataset (left) and COCO (Chen et al., 2015) (right). Visual News pro- vides more informative captions with name entities, whereas COCO contains more generic captions. annotated captions, these captions were written so that they are descriptive rather than interpretative, and referents to objects are generic rather than spe- cific. For example, a caption such as “A bunch of people who are holding red umbrellas.” properly describes the image at some level to the right in Figure 1, but it fails to capture the higher level sit- uation that is taking place in this picture i.e. “why are people gathering with red umbrellas and what role do they …

Content truncated to 500 words