L2C: Describing Visual Differences Needs Semantic Understanding of Individuals

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2315–2320 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2315 L2C: Describing Visual Differences Needs Semantic Understanding of Individuals An Yan♦, Xin Eric Wang ♠, Tsu-Jui Fu♣, William Yang Wang♣ ♦UC San Diego, ♠UC Santa Cruz, ♣UC Santa Barbara ayan@ucsd.edu, xwang366@ucsc.edu, {tsu-juifu,william}@cs.ucsb.edu Abstract Recent advances in language and vision push forward the research of captioning a single im- age to describing visual differences between image pairs. Suppose there are two images, I1 and I2, and the task is to generate a de- scription W1,2 comparing them, existing meth- ods directly model ⟨I1, I2⟩ → W1,2 mapping without the semantic understanding of individ- uals. In this paper, we introduce a Learning- to-Compare (L2C) model, which learns to understand the semantic structures of these two images and compare them while learn- ing to describe each one. We demonstrate that L2C benefits from a comparison between explicit semantic representations and single- image captions, and generalizes better on the new testing image pairs. It outperforms the baseline on both automatic evaluation and hu- man evaluation for the Birds-to-Words dataset. 1 Introduction The task of generating textual descriptions of im- ages tests a machine’s ability to understand visual data and interpret it in natural language. It is a fun- damental research problem lying at the intersection of natural language processing, computer vision, and cognitive science. For example, single-image captioning (Farhadi et al., 2010; Kulkarni et al., 2013; Vinyals et al., 2015; Xu et al., 2015) has been extensively studied. Recently, a new intriguing task, visual compari- son, along with several benchmarks (Jhamtani and Berg-Kirkpatrick, 2018; Tan et al., 2019; Park et al., 2019; Forbes et al., 2019) has drawn increasing at- tention in the community. To complete the task and generate comparative descriptions, a machine should understand the visual differences between a pair of images (see Figure 1). Previous meth- ods (Jhamtani and Berg-Kirkpatrick, 2018) often consider the pair of pre-trained visual features such vs This b whit animal1 has a medium sized dark beak, a white breast and grey wings. animal2 has a white breast with brown wings and tail, black eyes and a brown head . vs Figure 1: Overview of the visual comparison task and our motivation. The key is to understand both images and compare them. Explicit semantic structures can be compared between images and used to generate com- parative descriptions aligned to the image saliency. as the ResNet features (He et al., 2016) as a whole, and build end-to-end neural networks to predict the description of visual comparison directly. In contrast, humans can easily reason about the vi- sual components of a single image and describe the visual differences between two images based on their semantic understanding of each one. Humans do not need to look at thousands of image pairs to describe the difference of new image pairs, as they can leverage their understanding of single images for visual comparison. Therefore, …

Content truncated to 500 words