Linking Entities Across Images and Text

Proceedings of the 19th Conference on Computational Language Learning, pages 185–193, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Linking Entities Across Images and Text Rebecka Weegar DSV Stockholm University Kalle ˚Astr¨om Dept. of Mathematics Lund University rebeckaw@dsv.su.se, kalle@maths.lth.se, Pierre.Nugues@cs.lth.se Pierre Nugues Dept. of Computer Science Lund University Abstract This paper describes a set of methods to link entities across images and text. As a corpus, we used a data set of images, where each image is commented by a short caption and where the regions in the im- ages are manually segmented and labeled with a category. We extracted the entity mentions from the captions and we com- puted a semantic similarity between the mentions and the region labels. We also measured the statistical associations be- tween these mentions and the labels and we combined them with the semantic sim- ilarity to produce mappings in the form of pairs consisting of a region label and a caption entity. In a second step, we used the syntactic relationships between the mentions and the spatial relationships between the regions to rerank the lists of candidate mappings. To evaluate our methods, we annotated a test set of 200 images, where we manually linked the im- age regions to their corresponding men- tions in the captions. Eventually, we could match objects in pictures to their correct mentions for nearly 89 percent of the seg- ments, when such a matching exists. 1 Introduction Linking an object in an image to a mention of that object in an accompanying text is a challenging task, which we can imagine useful in a number of settings. It could, for instance, improve im- age retrieval by complementing the geometric re- lationships extracted from the images with textual descriptions from the text. A successful mapping would also make it possible to translate knowledge and information across image and text. In this paper, we describe methods to link men- tions of entities in captions to labeled image seg- ments and we investigate how the syntactic struc- ture of a caption can be used to better understand the contents of an image. We do not address the closely related task of object recognition in the im- ages. This latter task can be seen as a complement to entity linking across text and images. See Rus- sakovsky et al. (2015) for a description of progress and results to date in object detection and classifi- cation in images. 2 An Example Figure 1 shows an example of an image from the Segmented and Annotated IAPR TC-12 data set (Escalantea et al., 2010). It has four regions la- beled cloud, grass, hill, and river, and the caption: a flat landscape with a dry meadow in the foreground, a lagoon behind it and many clouds in the sky containing mentions of five entities that we iden- tify with the words meadow, landscape, lagoon, cloud, and sky. A correct association of the men- tions in the caption to the image regions would Figure 1: …

Content truncated to 500 words