Transformer-Exclusive Cross-Modal Representation for Vision and Language

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2719–2725 August 1–6, 2021. ©2021 Association for Computational Linguistics 2719 Transformer-Exclusive Cross-Modal Representation for Vision and Language Andrew Shin Sony Group Corporation andrew.shin@sony.com Takuya Narihira Sony Group Corporation takuya.narihira@sony.com Abstract Ever since the advent of deep learning, cross- modal representation learning has been dom- inated by the approaches involving convolu- tional neural networks for visual representa- tion and recurrent neural networks for lan- guage representation. Transformer architec- ture, however, has rapidly taken over the re- current neural networks in natural language processing tasks, and it has also been shown that vision tasks can be handled with trans- former architecture, with compatible perfor- mance to convolutional neural networks. Such results naturally lead to speculation upon the possibility of tackling cross-modal representa- tion for vision and language exclusively with transformer. This paper examines transformer- exclusive cross-modal representation to ex- plore such possibility, demonstrating its poten- tials as well as discussing its current limita- tions and its prospects. 1 Introduction While early cross-modal models handled visuolin- guistic tasks with template-based methods (Barbu et al., 2012; Elliott and Keller, 2013), or as a re- trieval model (Farhadi et al., 2010; Ordonez et al., 2011), the advent of deep learning introduced end-to-end learning models for cross-modal tasks, in which convolutional neural networks (CNNs) (Krizhevsky et al., 2012) are employed for vision representation, whereas recurrent neural networks (RNNs), such as LSTM (Hochreiter and Schmidhu- ber, 1997) or GRU (Cho et al., 2014), are employed for language representation. While a plethora of variations exist, most models proposed in the past few years have invariably relied on the CNN-RNN approach. Such standardized scheme, however, started to change with the introduction of transformer archi- tecture based on multi-head attention mechanism (Vaswani et al., 2017), which rapidly started to achieve state-of-the-art performance in natural lan- guage processing (NLP) (Peters et al., 2018; Dai et al., 2019; Yang et al., 2019) and speech recog- nition domains (Dong et al., 2018; Wang et al., 2020b), frequently outperforming RNNs. Further- more, large-scale models based on transformer architecture, such as BERT (Devlin et al., 2019) or GPT-3 (Brown et al., 2020), started to appear, demonstrating that pre-training a sufficiently large model with a very large amount of data results in strong performance with versatility for various downstream tasks. The success of transformer-based models in NLP and speech recognition naturally led to its adapta- tion in cross-modal tasks. (Lu et al., 2019) pro- posed ViLBERT, a pioneering BERT-inspired work that proposed to tokenize the images for compati- bility with transformer architecture, and also to ex- tend the pre-training objectives of BERT to reflect the nature of cross-modality. Many other cross- modal models followed, but mostly with similar approaches for image tokenization and pre-training objectives. This line of transformer-based cross- modal works described above, however, still heav- ily relied on CNN-based models, such as Faster R-CNN (Ren et al., 2015), to extract features from images, and the application of transformer was mostly limited to language representation. …

Content truncated to 500 words