Semantically Distributed Robust Optimization for Vision-and-Language Inference

Findings of the Association for Computational Linguistics: ACL 2022, pages 1493 - 1513 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Semantically Distributed Robust Optimization for Vision-and-Language Inference Tejas Gokhale∗, Abhishek Chaudhary∗, Pratyay Banerjee, Chitta Baral, and Yezhou Yang Arizona State University {tgokhale, achaud39, pbanerj6, chitta, yz.yang}@asu.edu Abstract Analysis of vision-and-language models has revealed their brittleness under linguistic phe- nomena such as paraphrasing, negation, tex- tual entailment, and word substitutions with synonyms or antonyms. While data augmen- tation techniques have been designed to miti- gate against these failure modes, methods that can integrate this knowledge into the training pipeline remain under-explored. In this paper, we present SDRO†, a model-agnostic method that utilizes a set linguistic transformations in a distributed robust optimization setting, along with an ensembling technique to lever- age these transformations during inference. Experiments on benchmark datasets with im- ages (NLVR2) and video (VIOLIN) demon- strate performance improvements as well as ro- bustness to adversarial attacks. Experiments on binary VQA explore the generalizability of this method to other V&L tasks. 1 Introduction “Does the text match the image?” – this simple question represents the Vision-and- Language Inference (VLI) task, as shown in Fig- ure 1. Image-text matching forms the backbone for V&L pre-training (Sun et al., 2019; Tan and Bansal, 2019; Lu et al., 2019) and has resulted in improve- ments in downstream tasks such as visual question answering, image retrieval, referring expressions, and visual commonsense reasoning. While natu- ral language inference (without visual inputs) has been extensively studied (Bowman et al., 2015; Williams et al., 2018; Khot et al., 2018; Demszky et al., 2018), VLI demands the additional capabil- ity of being grounded in the scene while under- standing semantics. Although pre-trained language models (PLMs) (Vaswani et al., 2017; Devlin et al., 2019; Raffel et al., 2020) have been useful for en- coding text into vector embeddings, recent find- ∗Equal Contribution †https://github.com/ASU-APG/VLI_SDRO At least one sail boat is parked on the left side of the dock. The left and right image contains the same number of sailboats sailing on the water. True False NLVR2 VIOLIN The phone rings, a man picks it up, and a woman slams her hand on the desk and demands the man give her the phone. The man realizes it is the woman's mother who is calling and he passes the phone back to the woman True False - Gavin Mitchell’s office. Rachen Green’s office. - Give me that phone - Hello this is Rachel Green. How can I help you? - Uh-huh. Okay then. I’ll pass you back to your son. - Hey, Mom. No that’s just my secretary. Figure 1: VLI models predict whether a sentence is True or False, given the visual input. (Top) sam- ple from NLVR2 with two images as input; (bottom) sample from VIOLIN with video and subtitles as input. ings point to undesirably high cosine similarity of two random words (Ethayarajh, 2019), the struggle with negation (Kassner and Schütze, 2020; Ettinger, 2020), and semantically equivalent adversarial ex- …

Content truncated to 500 words