Learning to Relate from Captions and Bounding Boxes

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6597–6603 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6597 Learning to Relate from Captions and Bounding Boxes Sarthak Garg⇤, Joel Ruben Antony Moniz⇤, Anshu Aviral⇤, Priyatham Bollimpalli⇤ School of Computer Science Carnegie Mellon University {sarthakg, jrmoniz, aanshu, pbollimp}@cs.cmu.edu Abstract In this work, we propose a novel approach that predicts the relationships between various en- tities in an image in a weakly supervised man- ner by relying on image captions and object bounding box annotations as the sole source of supervision. Our proposed approach uses a top-down attention mechanism to align en- tities in captions to objects in the image, and then leverage the syntactic structure of the cap- tions to align the relations. We use these alignments to train a relation classification net- work, thereby obtaining both grounded cap- tions and dense relationships. We demonstrate the effectiveness of our model on the Visual Genome dataset by achieving a recall@50 of 15% and recall@100 of 25% on the relation- ships present in the image. We also show that the model successfully predicts relations that are not present in the corresponding captions. 1 Introduction Scene graphs serve as a convenient representation to capture the entities in an image and the relation- ships between them, and are useful in a variety of settings (for example, Johnson et al. (2015); Ander- son et al. (2016); Liu et al. (2017)). While the last few years have seen considerable progress in classi- fying the contents of an image and segmenting the entities of interest without much supervision (He et al., 2017), the task of identifying and understand- ing the way in which entities in an image interact with each other without much supervision remains little explored. Recognizing relationships between entities is non-trivial because the space of possible relation- ships is immense, and because there are O(n2) relationships possible when n objects are present in an image. On the other hand, while image captions are easier to obtain, they are often not ⇤Equal Contribution completely descriptive of an image (Krishna et al., 2017). Thus, simply parsing a caption to extract relationships from them is likely to not sufficiently capture the rich content and detailed spatial rela- tionships present in an image. Since different images have different objects and captions, we believe it is possible to get the informa- tion that is not present in the caption of one image from other similar images which have the same objects and their captions. In this work, we thus aim to learn the relationships between entities in an image by utilizing only image captions and object locations as the source of supervision. Given that generating a good caption in an image requires one to understand the various entities and the relation- ships between them, we hypothesize that an image caption can serve as an effective weak supervisory signal for relationship prediction. 2 Related work The task of Visual Relationship Detection has …

Content truncated to 500 words