Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2034–2045, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2034 Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents Jack Hessel Lillian Lee Cornell UniversityDavid Mimno {jhessel, llee}@cs.cornell.edu mimno@cornell.edu Abstract Images and text co-occur constantly on the web, but explicit links between images and sentences (or other intra-document textual units) are often not present. We present al- gorithms that discover image-sentence rela- tionships without relying on explicit multi- modal annotation in training. We experi- ment on seven datasets of varying difficulty, ranging from documents consisting of groups of images captioned post hoc by crowdwork- ers to naturally-occurring user-generated mul- timodal documents. We find that a struc- tured training objective based on identifying whether collections of images and sentences co-occur in documents can suffice to predict links between specific sentences and specific images within the same document at test time. 1 Introduction Images and text act as natural complements on the modern web. News stories include photographs, product listings show multiple images providing detail for online shoppers, and Wikipedia pages include maps, diagrams, and pictures. But the ex- act matching between words and images is often left implicit. Algorithms that identify document- internal connections between specific images and specific passages of text could have both immedi- ate and long-term promise. On the user-experience front, alt-text for vision-impaired users could be produced automatically (Wu et al., 2017) via intra- document retrieval, and user interfaces could ex- plicitly link images to descriptive sentences, poten- tially improving the reading experience of sighted users. Also, in terms of improving other appli- cations, the text in multimodal documents can be viewed as a noisy form of image annotation: in- ferred image-sentence associations can serve as training pairs for vision models, particularly in do- mains lacking readily-available labeled data. Training Time: Document-level Co-occurrence Testing Time: Image/Sentence Link Prediction Great day at the park! Played frisbee with the dog. Won our ultimate game! Figure 1: At training time, we assume we are given a set of multi-image/multi-sentence documents. At test- time, we predict links between individual images and individual sentences within single documents. Because no explicit multimodal annotation is available at train- ing time, we refer to this task as unsupervised. In this work, we develop unsupervised models that learn to identify multimodal within-document links despite not having access to supervision at the individual image/sentence level during training. Rather, the training documents contain multiple images and multiple sentences1 that are not aligned, as illustrated in Figure 1. Our intra-document setting poses challenges be- yond those encountered in the usual cross-modal retrieval framework, wherein “documents” gener- ally consist of a single image associated with a single piece of text, e.g., an image caption. For the longer documents we consider, a sentence may have many corresponding images or no correspond- ing images, and vice versa. Furthermore, we expect that images within documents will be, …

Content truncated to 500 words