X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8785–8805, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8785 X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers Jaemin Cho1,2∗ Jiasen Lu1 Dustin Schwenk1 Hannaneh Hajishirzi1,3 Aniruddha Kembhavi1,3 Allen Institute for AI1 UNC Chapel Hill2 University of Washington3 jmincho@cs.unc.edu {jiasenl,dustins,hannah,anik}@allenai.org Code and Demo: https://prior.allenai.org/projects/x-lxmert Abstract Mirroring the success of masked language models, vision-and-language counterparts like VILBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family – LXMERT – finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with training refinements includ- ing: discretizing visual representations, using uniform masking with a large range of mask- ing ratios and aligning the right pre-training datasets to the right objectives which enables it to paint. X-LXMERT’s image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT. Finally, we demonstrate the generality of these training refinements by adding image generation capabilities into UNITER to produce X-UNITER. 1 Introduction The past year has seen a spate of BERT-style (De- vlin et al., 2019) transformer-based architectures (Lu et al., 2019; Chen et al., 2019; Li et al., 2019) proposed for vision-and-language tasks. These models are typically pre-trained on large image captioning corpora, extending ideas from masked language modeling to mask both the image and text modalities and produce state of the art results ∗This work was done as part of the Pre-Doctoral Young Investigator residency program at the Allen Institute for AI. on a variety of vision and language tasks includ- ing visual question answering, visual grounding and image retrieval. These impressive results as well as recent probing mechanisms (Ilharco et al., 2020) suggest that these models are able to capture a variety of semantics in images including objects, attributes and their relationships and ground these in natural language. While these models have been extensively eval- uated over several discriminative tasks, relatively little attention has been paid to their generative ca- pabilities. Bidirectional transformer models like BERT which exploit context preceding and follow- ing the current token are not explicitly designed for generation. Recent work for language-only trans- formers (Wang and Cho, 2019; Dong et al., 2019; Liao et al., 2020) adapt these models towards this capability using sampling procedures. Such tech- niques have also been adapted successfully for im- age captioning - inputting an image and sampling the textual side of the model to generate a relevant caption (Zhou et al., 2020). This begs the question: Can we go the other way and sample images from input pieces …

Content truncated to 500 words