TraVLR: Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3322–3347 May 2-6, 2023 ©2023 Association for Computational Linguistics TRAVLR: Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguisic Reasoning Keng Ji Chow♮λ∗ Samson Tan♮§∗ Min-Yen Kan♮ ♮Department of Computer Science, National University of Singapore λDepartment of English Language and Literature, National University of Singapore §AWS AI Research & Education kengjichow@u.nus.edu {samson.tmr, kanmy}@comp.nus.edu.sg Abstract Numerous visio-linguistic (V+L) representa- tion learning methods have been developed, yet existing datasets do not adequately eval- uate the extent to which they represent vi- sual and linguistic concepts in a unified space. We propose several novel evaluation settings for V+L models, including cross-modal trans- fer. Furthermore, existing V+L benchmarks often report global accuracy scores on the en- tire dataset, making it difficult to pinpoint the specific reasoning tasks that models fail and succeed at. We present TRAVLR, a syn- thetic dataset comprising four V+L reasoning tasks. TRAVLR’s synthetic nature allows us to constrain its training and testing distribu- tions along task-relevant dimensions, enabling the evaluation of out-of-distribution generali- sation. Each example in TRAVLR redundantly encodes the scene in two modalities, allowing either to be dropped or added during training or testing without losing relevant information. We compare the performance of four state-of- the-art V+L models, finding that while they perform well on test examples from the same modality, they all fail at cross-modal transfer and have limited success accommodating the addition or deletion of one modality. We re- lease TRAVLR as an open challenge for the research community.1 1 Introduction Research in psycholinguistics has found that hu- man processing of spatial words activates brain regions associated with the visual system (Tang et al., 2021), suggesting the latter’s involvement in processing linguistic input. It is reasonable to ex- pect multimodal neural models to resemble humans in being able to leverage capabilities in the visual domain to solving problems in the text domain, and vice versa. Following its recent success in the ∗Equal contribution 1Code and dataset available at https://github.com/ kengjichow/TraVLR. (a) A complete example for the spatiality task. (b) Possible directions of cross-modal transfer. Figure 1: An example from TRAVLR (a). Both im- age and caption fully represent the scene; either can be dropped during training/testing, enabling the evalua- tion of cross-modal transfer ability (b). text domain (Devlin et al., 2019), the pretraining– fine-tuning paradigm has been applied to the vision and text modalities to create unified visio-linguistic (V+L) representations. Just as pretrained multilin- gual models have been shown capable of zero-shot cross-lingual transfer on various NLP tasks (Con- neau et al., 2020), we may expect true V+L models to be capable of generalising to a modality not seen during fine-tuning. However, current approaches for benchmark- ing V+L models involve reporting global accu- racy scores on the entire dataset, rendering the specific sources of success and failure difficult to diagnose (Ribeiro et al., 2020; Goel et al., 2021). For instance, Visual Question Answering (VQA, Goyal et …

Content truncated to 500 words