CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 61–70 July 13-14, 2023 ©2023 Association for Computational Linguistics CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling Abhidip Bhattacharyya and Martha Palmer and Christoffer Heckman University of Colorado Boulder firstname.lastname@colorado.edu Abstract Automatic image comprehension is an impor- tant yet challenging task that includes identi- fying actions in an image and corresponding action participants. Most current approaches to this task, now termed Grounded Situation Recognition (GSR), start by predicting a verb that describes the action and then predict the nouns that can participate in the action as argu- ments to the verb. This problem formulation limits each image to a single action even though several actions could be depicted. In contrast, text-based Semantic Role Labeling (SRL) aims to label all actions in a sentence, typically re- sulting in at least two or three predicate argu- ment structures per sentence. We hypothesize that expanding GSR to follow the more liberal SRL text-based approach to action and partici- pant identification could improve image com- prehension results. To test this hypothesis and to preserve generalization capabilities, we use general-purpose vision and language compo- nents as a front-end. This paper presents our results, a substantial 28.6 point jump in per- formance on the SWiG dataset, which confirm our hypothesis. We also discuss the benefits of loosely coupled broad-coverage off-the-shelf components which generalized well to out of domain images, and can decrease the need for manual image semantic role annotation. 1 Introduction Automatic image comprehension can positively contribute to many modern applications, such as description generation, cross-modal retrieval, and human-robot interaction. To comprehend an image it is important to identify the action(s) and partic- ipants in the action such asan agent (who is per- forming the action), a patient (who is being affected by the action), and an instrument. To address this problem (Yatskar et al., 2016b; Pratt et al., 2020) proposed the task of grounded situation recognition (GSR). Many approaches (Pratt et al., 2020; Cooray et al., 2020; Cho et al., 2021) have been proposed a. drinking c. drinking d. wearing b. manipulation (holding) agent clothing agent item container agent item container wearer Figure 1: a. depicts a GT example from SWiG where the man is the agent of drinking. b., c., and d. show frames extracted by our method. Bounding boxes depict grounding and role annotation for each frame. to perform the task of GSR. Most of these frame- works have two steps: in the first step verbs are predicted, and in the second step nouns and roles are predicted in an auto-regressive manner. Some other methods deployed include another layer to refine the quality of detection (Cho et al., 2021; Wei et al., 2021; Cheng et al., 2022). One fundamental limitation of these models de- rives from the problem formulation. In the cur- rent formulation, verb frames would compete for an image, limiting the expressiveness of the im- age’s semantic representation. In reality, various actions can co-exist in …

Content truncated to 500 words