Image Position Prediction in Multimodal Documents

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4265–4274 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4265 Image Position Prediction in Multimodal Documents Masayasu Muraoka†, Ryosuke Kohita†, Etsuko Ishii‡ †IBM Research – Tokyo, Japan ‡Hong Kong University of Science and Technology, Hong Kong mmuraoka@jp.ibm.com, kohi@ibm.com, eishii@connect.ust.hk Abstract Conventional multimodal tasks, such as caption generation and visual question answering, have allowed machines to understand an image by describing or being asked about it in natural language, often via a sentence. Datasets for these tasks contain a large number of pairs of an image and the corresponding sentence as an instance. However, a real multimodal document such as a news article or Wikipedia page consists of multiple sentences with multiple images. Such documents require an advanced skill of jointly considering the multiple texts and multiple images, beyond a single sentence and image, for the interpretation. Therefore, aiming at building a system that can understand multimodal documents, we propose a task called image position prediction (IPP). In this task, a system learns plausible positions of images in a given document. To study this task, we automatically constructed a dataset of 66K multimodal documents with 320K images from Wikipedia articles. We conducted a preliminary experiment to evaluate the performance of a current multimodal system on our task. The experimental results show that the system outperformed simple baselines while the performance is still far from human performance, which thus poses new challenges in multimodal research. Keywords: resource creation, multimodal document understanding, vision and language 1. Introduction Connecting the modalities of vision and language is one of the most ambitious goals in computer vision and natural language processing. A number of multimodal tasks has been proposed and made a great progress to this end. For example, caption generation (Lin et al., 2014) and visual question answering (VQA) (Antol et al., 2015; Agrawal et al., 2017) are well-known conventional multimodal tasks that have received much attention in recent years (Hossain et al., 2019; Kafle and Kanan, 2017; Wu et al., 2017). Cap- tion generation aims to generate a caption to describe the contents of an input image with a natural language sen- tence while the goal of VQA is to answer a question written in natural language about a given image. These tasks can be solved by interpreting an image and the corresponding short text such as a caption or question. However, real multimodal documents consist of multiple sentences and multiple images as seen anywhere. For ex- ample, newswire articles contain photographs of events, cooking recipes include pictures of intermediate stages of cooking, and Wikipedia articles contain various types of images showing people, buildings, scenery, and various ob- jects. Being placed at appropriate positions in such a docu- ment, images help people understand the whole document by naturally considering the correspondence of the images and document. Unfortunately, existing multimodal tasks have not dealt with this high-level correspondence, because an instance in a typical dataset often consists …

Content truncated to 500 words