Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4944–4954 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4944 Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding Abhidip Bhattacharyya, Cecilia Mauceri, Martha Palmer, Christoffer Heckman University of Colorado Boulder firstname.lastname@colorado.edu Abstract As vision processing and natural language processing continue to advance, there is increasing interest in multimodal applica- tions, such as image retrieval, caption generation, and human-robot interaction. These tasks require close alignment between the information in the images or video and text. In this paper, we present a new semantic labeling method that combines state-of-the-art semantic annotation for language with the bounding boxes of corresponding images. This richer multimodal labeling supports cross-modal inference for applications in which such alignment is useful. Our semantic representations, developed in the natural language processing community, abstract away from the surface structure of the sentence, focusing on specific actions and the roles of their participants. This level of abstraction maps well to the objects, actions, and relationships visible in images. We utilize these representations in the form of semantic role labels in the captions and the images and demonstrate improvements in standard tasks such as image retrieval. The potential contributions of these additional labels is evaluated using a role-aware retrieval system based on graph convolutional and recurrent neural networks. The addition of semantic roles into this system provides a significant increase in capability and greater flexibility for these tasks, and could be extended to state-of-the-art techniques relying on transformers with larger amounts of annotated data. Keywords: cross-modal retrieval, semantic role labeling, 1. Introduction Vision processing is making exciting advances and per- formance is rapidly improving for tasks such as caption generation, question answering and retrieval. In paral- lel, NLP is making corresponding advances, fueled by both vector representations and rich semantic represen- tations (Wang et al., 2021). In this paper, we explore the benefits of combining the rich semantic represen- tations of NLP with image bounding boxes. A con- gruent description of an image should be semantically grounded by the objects presented in the image. There- fore composing a concise image description requires focusing on a few contextually salient entities, proper- ties, or events, instead of being exhaustive. For exam- ple, in Figure 1, the example captions focus on the man. However, a description that focused on the rifle or on the dog would be equally valid. Currently, image re- trieval systems can retrieve relevant results for diverse input, but they do not provide a way to intentionally inject variety into the search results by specifying a de- sired focus. Semantic role labeling (SRL) (Palmer et al., 2005) is a form of semantic parsing developed for natural lan- guage processing that conveys knowledge about who is doing what to whom when as predicate-argument struc- tures. In other words, given an action in a sentence, one needs to know who is performing the action (the agent), who is affected by the action …

Content truncated to 500 words