The Connection between the Text and Images of News Articles: New Insights for Multimedia Analysis

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4343–4351 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4343 The Connection between the Text and Images of News Articles: New Insights for Multimedia Analysis Nelleke Oostdijk1, Hans van Halteren1, Erkan Bașar2, Martha Larson1 1CLS, Radboud University, Nijmegen 2FloodTags, The Hague {n.oostdijk, hvh, m.larson}@let.ru.nl, basar@floodtags.com Abstract We report on a case study of text and images that reveals the inadequacy of simplistic assumptions about their connection and interplay. The context of our work is a larger effort to create automatic systems that can extract event information from online news articles about flooding disasters. We carry out a manual analysis of 1,000 articles containing a keyword related to flooding. The analysis reveals that the articles in our data set cluster into seven categories related to different topical aspects of flooding, and that the images accompanying the articles cluster into five categories related to the content they depict. The results demonstrate that flood-related news articles do not consistently report on a single, currently unfolding flooding event and we should also not assume that a flood-related image will directly relate to a flooding-event described in the corresponding article. In particular, spatiotemporal distance is important. We validate the manual analysis with an automatic classifier demonstrating the technical feasibility of multimedia analysis approaches that admit more realistic relationships between text and images. In sum, our case study confirms that closer attention to the connection between text and images has the potential to improve the collection of multimodal information from news articles. Keywords: Corpus (Creation, Annotation, etc.), Multimedia Document Processing, Tools, Systems, Applications 1. Introduction Online news is a natural source of information about current events. News articles consist of components including headlines, article text, images, and image captions. In order to build a system that can automatically extract event information from online news, it is important to understand the connection between these components. In this paper, we report the results of a case study that we carried out in order to gain a better understanding of this connection. Our larger goal, which is the wider context for this paper, is to create a system that can automatically collect and aggregate information about the impact of flooding from online sources. Such information supports disaster managers in monitoring and responding to flooding events as they evolve, and also helps scientists gain a better understanding of flooding by allowing them to analyze historical patterns (FloodTags, 2019). Event information from online sources provides an important complement to information that can be derived from remote sensing imagery. Specifically, it is not dependent on factors such as the periodicity of the satellite or cloud-cover conditions. Further, it reflects not only the development of the disaster, but also the specific impact that the disaster is having on people. Our aim is to fully take advantages of both the text and the image information in online sources, and for this reason we are focused on …

Content truncated to 500 words