COFAR: Commonsense and Factual Reasoning in Image Search

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1185–1199 November 20–23, 2022. ©2022 Association for Computational Linguistics 1185 COFAR: Commonsense and Factual Reasoning in Image Search Prajwal Gatti1, Abhirama Subramanyam Penamakuri1, Revant Teotia2,*, Anand Mishra1, Shubhashis Sengupta3, Roshni Ramnani3 1Indian Institute of Technology Jodhpur, 2Columbia University, 3Accenture Labs {pgatti, penamakuri.1, mishra}@iitj.ac.in, rt2819@columbia.edu {shubhashis.sengupta, roshni.r.ramnani}@accenture.com Figure 1: Consider the following two natural language queries shown in (a). Retrieving images relevant to these queries (shown using a green bounding box) requires a model that has the ability to interpret images beyond just what is visually apparent, such as interpreting – who are customers vs. who are tourists? Who are waiting to buy vs. who are going to see? in other words, visual commonsense. Additionally, the model would need to interpret facts or world knowledge, such as Häagen-Dazs is an ice cream brand and the Taj Mahal in India is an example of Mughal architecture. This can be enabled by linking visual entities in the image to an encyclopedic knowledge source such as Wikipedia. Our work presents such a model, namely KRAMT. Abstract One characteristic that makes humans supe- rior to modern artificially intelligent models is the ability to interpret images beyond what is visually apparent. Consider the following two natural language search queries – (i) “a queue of customers patiently waiting to buy ice cream" and (ii) “a queue of tourists going to see a famous Mughal architecture in India." Interpreting these queries requires one to rea- son with (i) Commonsense such as interpret- ing people as customers or tourists, actions as waiting to buy or going to see; and (ii) Fact or world knowledge associated with named vi- sual entities, for example, whether the store in the image sells ice cream or whether the landmark in the image is a Mughal architecture located in India. Such reasoning goes beyond just visual recognition. To enable both com- monsense and factual reasoning in the image search, we present a unified framework, namely *This work was done while Revant Teotia was affiliated with Indian Institute of Technology Jodhpur. Knowledge Retrieval-Augmented Multimodal Transformer (KRAMT), that treats the named visual entities in an image as a gateway to ency- clopedic knowledge and leverages them along with natural language query to ground relevant knowledge. Further, KRAMT seamlessly in- tegrates visual content and grounded knowl- edge to learn alignment between images and search queries. This unified framework is then used to perform image search requiring com- monsense and factual reasoning. The retrieval performance of KRAMT is evaluated and com- pared with related approaches on a new dataset we introduce – namely COFAR. We make our code and dataset available at https:// vl2g.github.io/projects/cofar. 1 Introduction Retrieving relevant images for a natural language query has been an exciting field of research in the vision-and-language community (Johnson et al., 2015; Wang et al., 2016a, 2020). Most of the avail- …

Content truncated to 500 words