LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5100–5111, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5100 LXMERT: Learning Cross-Modality Encoder Representations from Transformers Hao Tan Mohit Bansal UNC Chapel Hill {haotan, mbansal}@cs.unc.edu Abstract Vision-and-language reasoning requires an un- derstanding of visual concepts, language se- mantics, and, most importantly, the align- ment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Represen- tations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an ob- ject relationship encoder, a language encoder, and a cross-modality encoder. Next, to en- dow our model with the capability of con- necting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse rep- resentative pre-training tasks: masked lan- guage modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image ques- tion answering. These tasks help in learn- ing both intra-modality and cross-modality re- lationships. After fine-tuning from our pre- trained parameters, our model achieves the state-of-the-art results on two visual ques- tion answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre- trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demon- strate detailed ablation studies to prove that both our novel model components and pre- training strategies significantly contribute to our strong results.1 1 Introduction Vision-and-language reasoning requires the un- derstanding of visual contents, language seman- tics, and cross-modal alignments and relation- 1Published at EMNLP 2019. Code and pre-trained mod- els publicly available at: https://github.com/airsplay/lxmert ships. There has been substantial past works in separately developing backbone models with bet- ter representations for the single modalities of vi- sion and of language. For visual-content under- standing, people have developed several backbone models (Simonyan and Zisserman, 2014; Szegedy et al., 2015; He et al., 2016) and shown their ef- fectiveness on large vision datasets (Deng et al., 2009; Lin et al., 2014; Krishna et al., 2017). Pi- oneering works (Girshick et al., 2014; Xu et al., 2015) also show the generalizability of these pre- trained (especially on ImageNet) backbone mod- els by fine-tuning them on different tasks. In terms of language understanding, last year, we witnessed strong progress towards building a universal back- bone model with large-scale contextualized lan- guage model pre-training (Peters et al., 2018; Rad- ford et al., 2018; Devlin et al., 2019), which has improved performances on various tasks (Ra- jpurkar et al., 2016; Wang et al., 2018) to sig- nificant levels. Despite these influential single- modality works, large-scale pretraining and fine- tuning studies for the modality-pair of vision and language are still under-developed. Therefore, we present one of the first works in building a pre-trained vision-and-language cross- modality framework and show its strong perfor- mance on …

Content truncated to 500 words