Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2189–2196 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2189 Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models Taichi Iki1,2 and Akiko Aizawa1,2 1National Institute of Informatics, Chiyoda-ku, Tokyo, Japan 2Graduate University for Advanced Studies, Hayama, Kanagawa, Japan {iki,aizawa}@nii.ac.jp Abstract A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre- training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the orig- inal language model. To see how well this is achieved, we propose to evaluate V&L mod- els using an NLU benchmark (GLUE). We compare five V&L models, including single- stream and dual-stream models, trained with the same pre-training. Dual-stream mod- els, with their higher modality independence achieved by approximately doubling the num- ber of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much dif- ferent than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effec- tive method for improving the maintenance of language knowledge in V&L extensions. 1 Introduction Pre-trained vision-and-language (V&L) models im- prove the performance of tasks that require an un- derstanding of the V&L grounding, including vi- sual question answering (Antol et al., 2015), re- ferring expression comprehension (Kazemzadeh et al., 2014), and image-text matching (ITM) (Suhr et al., 2019). Recent V&L tasks, such as multi- modal reading comprehension (Kembhavi et al., 2017; Yagcioglu et al., 2018; Hannan et al., 2020; Tanaka et al., 2021) and dialogue (Ilinykh et al., 2019; Haber et al., 2019; Udagawa and Aizawa, 2019), require a deeper NLU as well as the ground- ing. Extending pre-trained language models (LMs) is an option for those tasks as this allows V&L models to inherit language knowledge from their source LMs. The typical extending consists of vi- sual pre-training and structure such as the stream type; the single-stream inserts vision tokens into the input sequence of the LM, and the dual-stream uses another sequence for early visual encoding. One of the remaining challenges is to understand how such extensions affect the pre-trained language knowledge. For example, Lu et al. (2019) proposed the dual-stream model where part of the goal was to protect the learned LMs. The authors focused on evaluation with V&L tasks and did not evaluate their models with language-only tasks. Cao et al. (2020) evaluated the extent of language knowl- edge loss in the single/dual-stream models against the source LM using language-only tasks. How- ever, the difference between single-stream and dual- stream models was unclear because the pre-training was also different in their models. In this paper, we investigate the effect of visual extensions in V&L models on language-only tasks1. Bugliarello et al. (2020) proposed a framework …

Content truncated to 500 words