There’s a Time and Place for Reasoning Beyond the Image

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1138 - 1149 May 22-27, 2022 c⃝2022 Association for Computational Linguistics There’s a Time and Place for Reasoning Beyond the Image Xingyu Fu1, Ben Zhou1∗, Ishaan Preetam Chandratreya2∗, Carl Vondrick2, Dan Roth1 1University of Pennsylvania 2Columbia University {xingyuf2, xyzhou, danroth}@seas.upenn.edu {ipc2107, cv2428}@columbia.edu Abstract Images are often more significant than only the pixels to human eyes, as we can infer, asso- ciate, and reason with contextual information from other sources to establish a more com- plete picture. For example, in Figure 1, we can find a way to identify the news articles re- lated to the picture through segment-wise un- derstandings of the signs, the buildings, the crowds, and more. This reasoning could pro- vide the time and place the image was taken, which will help us in subsequent tasks, such as automatic storyline construction, correction of image source in intended effect photographs, and upper-stream processing such as image clustering for certain location or time. In this work, we formulate this problem and introduce TARA: a dataset with 16k images with their associated news, time, and loca- tion, automatically extracted from New York Times1 (NYT), and an additional 61k exam- ples as distant supervision from WIT (Srini- vasan et al., 2021). On top of the extractions, we present a crowdsourced subset in which we believe it is possible to find the images’ spatio- temporal information for evaluation purpose. We show that there exists a 70% gap between a state-of-the-art joint model and human perfor- mance, which is slightly filled by our proposed model that uses segment-wise reasoning, moti- vating higher-level vision-language joint mod- els that can conduct open-ended reasoning with world knowledge. The data and code are publicly available at https://github. com/zeyofu/TARA. 1 Introduction Vision and language are two of most important in- formation sources, and the fact that humans reason jointly with both sources has motivated artificial intelligence research to consider visually-grounded ∗ Both authors contributed equally to this work. 1https://developer.nytimes.com/docs/ archive-product/1/overview Figure 1: This is an image from the New York Times. Can you tell the time and location when it was taken? language understanding. Most work in this area has focused on reasoning with local evidence (Suhr et al., 2019; Hudson and Manning, 2019; Lu et al., 2020; Liu et al., 2021), e.g. asking factoid ques- tions such as the colors or shapes of objects and numbers of people, yet very few works (Cui et al., 2021) encourage open-ended reasoning where a model needs to look beyond task inputs. However, humans can relate visual cues to corresponding con- textual information that could be multi-modal, and draw on background knowledge when interpreting and grounding images. For example, as Figure 1 shows, people that are familiar with the news can infer that the location is Times Square through the iconic screen panels, and further estimate the pe- riod of time by looking at the crowds and the signs. And, this can be done …

Content truncated to 500 words