Expressing Visual Relationships via Language

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1873–1883 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1873 Expressing Visual Relationships via Language Hao Tan1, Franck Dernoncourt2, Zhe Lin2, Trung Bui2, Mohit Bansal1 1UNC Chapel Hill 2Adobe Research {haotan, mbansal}@cs.unc.edu, {dernonco, zlin, bui}@adobe.com Abstract Describing images with text is a fundamen- tal problem in vision-language research. Cur- rent studies in this domain mostly focus on single image captioning. However, in vari- ous real applications (e.g., image editing, dif- ference interpretation, and retrieval), generat- ing relational captions for two images, can also be very useful. This important problem has not been explored mostly due to lack of datasets and effective models. To push for- ward the research in this direction, we first introduce a new language-guided image edit- ing dataset that contains a large number of real image pairs with corresponding editing in- structions. We then propose a new relational speaker model based on an encoder-decoder architecture with static relational attention and sequential multi-head attention. We also ex- tend the model with dynamic relational atten- tion, which calculates visual alignment while decoding. Our models are evaluated on our newly collected and two public datasets con- sisting of image pairs annotated with relation- ship sentences. Experimental results, based on both automatic and human evaluation, demon- strate that our model outperforms all baselines and existing methods on all the datasets.1 1 Introduction Generating captions to describe natural images is a fundamental research problem at the intersection of computer vision and natural language process- ing. Single image captioning (Mori et al., 1999; Farhadi et al., 2010; Kulkarni et al., 2011) has many practical applications such as text-based im- age search, photo curation, assisting of visually- impaired people, image understanding in social 1Our data and code are publicly available at: https://github.com/airsplay/ VisualRelationships Remove the people from the picture. Relational Speaker Figure 1: An example result of our method showing the input image pair from our Image Editing Request dataset, and the output instruction predicted by our re- lational speaker model trained on the dataset. media, etc. This task has drawn significant at- tention in the research community with numerous studies (Vinyals et al., 2015; Xu et al., 2015; An- derson et al., 2018), and recent state of the art methods have achieved promising results on large captioning datasets, such as MS COCO (Lin et al., 2014). Besides single image captioning, the com- munity has also explored other visual captioning problems such as video captioning (Venugopalan et al., 2015; Xu et al., 2016), and referring expres- sions (Kazemzadeh et al., 2014; Yu et al., 2017). However, the problem of two-image captioning, especially the task of describing the relationships and differences between two images, is still under- explored. In this paper, we focus on advanc- ing research in this challenging problem by intro- ducing a new dataset and proposing novel neural relational-speaker models.2 To the best of our knowledge, Jhamtani and Berg-Kirkpatrick (2018) is the only public …

Content truncated to 500 words