Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6555–6565 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6555 Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions Arjun R. Akula1∗, Spandana Gella2, Yaser Al-Onaizan2, Song-Chun Zhu1, Siva Reddy3 1UCLA Center for Vision, Cognition, Learning, and Autonomy, 2Amazon AI 3Facebook CIFAR AI Chair, Mila; McGill University aakula@ucla.edu, sgella@amazon.com, onaizan@amazon.com, sczhu@stat.ucla.edu, siva.reddy@mila.quebec Abstract Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target ob- ject, the word order doesn’t matter. To mea- sure the true progress of existing models, we split the test set into two sets, one which re- quires reasoning on linguistic structure and the other which doesn’t. Additionally, we create an out-of-distribution dataset Ref-Adv by ask- ing crowdworkers to perturb in-domain exam- ples such that the target object changes. Us- ing these datasets, we empirically show that existing methods fail to exploit linguistic struc- ture and are 12% to 23% lower in perfor- mance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of- the-art model for this task. Our datasets are publicly available at https://github.com/ aws/aws-refcocog-adv. 1 Introduction Visual referring expression recognition is the task of identifying the object in an image referred by a natural language expression (Kazemzadeh et al., 2014; Nagaraja et al., 2016; Mao et al., 2016; Hu et al., 2016). Figure 1 shows an example. This task has drawn much attention due to its ability to test a model’s understanding of natural lan- guage in the context of visual grounding and its application in downstream tasks such as image re- trieval (Young et al., 2014) and question answer- ing (Antol et al., 2015; Zhu et al., 2016). To track ∗Work done in part while AA was intern at Amazon AI. Model pastry on the plate next to a blue fork a blue fork next to the pastry plate r1 r2 r1 r1 Original Expression Adversarial Modification Input Image Region Proposals (r1, r2) Model Figure 1: An example of the visual referring expression recognition task. If the word pastry is present in the referring expression, models prefer the bounding box r1 (highlighted in green) irrespective of the change in linguistic structure (word order). progress on this task, various datasets have been proposed, in which real world images are anno- tated by crowdsourced workers (Kazemzadeh et al., 2014; Mao et al., 2016). Recently, neural mod- els have achieved tremendous progress on these datasets (Yu et al., 2018; Lu et al., 2019). However, multiple studies have suggested that these models could be exploiting strong biases in these …

Content truncated to 500 words