1 Million Captioned Dutch Newspaper Images

1 Million Captioned Dutch Newspaper Images Desmond Elliott∗† and Martijn Kleppe‡ ∗ILLC, University of Amsterdam; †CWI; ‡Erasmus University Rotterdam d.elliott@uva.nl, kleppe@eshcc.eur.nl Abstract Images naturally appear alongside text in a wide variety of media, such as books, magazines, newspapers, and in online articles. This type of multi-modal data offers an interesting basis for vision and language research but most existing datasets use crowdsourced text, which removes the images from their original context. In this paper, we introduce the KBK-1M dataset of 1.6 million images in their original context, with co-occurring texts found in Dutch newspapers from 1922 - 1994. The images are digitally scanned photographs, cartoons, sketches, and weather forecasts; the text is generated from OCR scanned blocks. The dataset is suitable for experiments in automatic image captioning, image–article matching, object recognition, and data-to-text generation for weather forecasting. It can also be used by humanities scholars to analyse photographic style changes, the representation of people and societal issues, and new tools for exploring photograph reuse via image-similarity-based search. Keywords: Digital Humanities, Digitised Newspapers, Language and Vision 1. Introduction Vision and language datasets typically consist of high- quality born-digital photographs paired with crowd- sourced descriptions (Rashtchian et al., 2010; Hodosh et al., 2013a; Young et al., 2014; Chen et al., 2015b). Descrip- tions obtained through crowdsourcing have several advan- tages: they usually describe what can be seen in the image, reducing the difficulty of visually grounding the words in the image, and it is easy to quickly collect multiple refer- ence texts. However, crowdsourcing takes an image outside its original context, rendering the intentions of the photog- rapher inaccessible. For example, does Figure 1 depict the world-famous Notre Dame Cathedral in Paris or automo- biles waiting at a traffic light? It is difficult to be sure, but crowdsourcing affects our ability to situate the image in a wider context. An alternative to crowdsourced descriptions is sourcing images that naturally co-occur with text, such as on photo sharing sites, social media, or in newspapers (Feng and Lapata, 2008; Ordonez et al., 2011; Chen et al., 2015a). The captions in these datasets often refer to specific people (politicians, movie stars, etc.), emotions, or places (Paris, my back garden, etc.) that are difficult to ground against the images. In this paper we introduce the KBK-1M dataset of 1.6 mil- lion captioned newspaper images. Contrary to other vi- sual datasets such as WikiMedia Commons, the KBK-1M dataset contains images and captions that have been cre- ated by professional photographers and journalists and they are situated in their original context in the newspaper. The dataset covers 72 years of printed newspapers from a va- riety of media outlets, giving a broad topical scope to the collection. The images are not digital-born, which causes interesting and unusual noise artifacts due to the age and quality of scanning printed paper. The images cover black- /white and colour photographs, sketches, line-drawn car- toons, and line-drawn weather forecasts; see Figure 3 for examples from each category. The image captions were ex- …

Content truncated to 500 words