Distilled Dual-Encoder Model for Vision-Language Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8901 - 8913 December 7-11, 2022 ©2022 Association for Computational Linguistics Distilled Dual-Encoder Model for Vision-Language Understanding Zekun Wang†∗, Wenhui Wang‡, Haichao Zhu†, Ming Liu†♮, Bing Qin†♮, Furu Wei‡ †Harbin Institute of Technology, Harbin, China ‡Microsoft Research, Beijing, China ♮Peng Cheng Laboratory, Shenzhen, China {zkwang,hczhu,mliu,qinb}@ir.hit.edu.cn {wenwan,fuwei}@microsoft.com Abstract On vision-language understanding (VLU) tasks, fusion-encoder vision-language models achieve superior results but sacrifice efficiency because of the simultaneous encoding of im- ages and text. On the contrary, the dual-encoder model that separately encodes images and text has the advantage in efficiency, while failing on VLU tasks due to the lack of deep cross-modal interactions. To get the best of both worlds, we propose DIDE1, a framework that distills the knowledge of the fusion-encoder teacher model into the dual-encoder student model. Since the cross-modal interaction is the key to the superior performance of teacher model but is absent in the student model, we encourage the student not only to mimic the predictions of teacher, but also to calculate the cross-modal at- tention distributions and align with the teacher. Experimental results demonstrate that DIDE is competitive with the fusion-encoder teacher model in performance (only a 1% drop) while enjoying 4× faster inference. Further analy- ses reveal that the proposed cross-modal atten- tion distillation is crucial to the success of our framework. 1 Introduction Vision-language understanding (VLU) tasks (e.g., visual reasoning (Suhr et al., 2019), visual entail- ment (Xie et al., 2019), visual question answer- ing (Goyal et al., 2017)) require the model to under- stand the cross-modal interactions between images and text. Various fusion-encoder vision-language pretrained models (Tan and Bansal, 2019; Chen et al., 2020; Zhang et al., 2021; Kim et al., 2021; Dou et al., 2022; Alayrac et al., 2022) are proposed for VLU tasks. As shown in Figure 1(a), these mod- els employ a Transformer (Vaswani et al., 2017) network as a cross-modal encoder to capture inter- actions between different modalities. Despite the ∗Contribution during internship at Microsoft Research. 1Our code and models will be publicly available at https: //github.com/kugwzk/DiDE. Cross-modal Transformer Encoder Visual Embedder/Encoder Textual Embedder/Encoder (a) Fusion-encoder Model Textual Encoder Visual Encoder MLP / Dot Product (b) Dual-encoder Model Figure 1: Illustration of two architectures of vision- language models. (a) Fusion-encoder models simul- taneously encode visual and textual inputs via modal- specific embedders/encoders and employ a cross-modal Transformer encoder to fuse representations. (b) Dual- encoder models encode images/text separately and adopt an extreme lightweight module (e.g., MLP) for cross-modal interactions. remarkable performance, the heavy cross-modal encoder remains an efficiency bottleneck due to the simultaneous encoding of images and text, limiting the application in practical scenarios with massive images or text. Therefore, it is crucial to find an approach to accelerate inference for VLU. We turn to explore the dual-encoder vision- language model (shown in Figure 1(b)), which encodes images and text separately and then ap- plies an extreme lightweight shallow module to model the interactions between modalities. The …

Content truncated to 500 words