Are Pretrained Convolutions Better than Pretrained Transformers?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4349–4359 August 1–6, 2021. ©2021 Association for Computational Linguistics 4349 Are Pre-trained Convolutions Better than Pre-trained Transformers? Yi Tay Google Research Mountain View, California yitay@google.com Mostafa Dehghani Google Research, Brain Team Amsterdam, Netherlands dehghani@google.com Jai Gupta Google Research Mountain View, California jaigupta@google.com Vamsi Aribandi∗ Google Research Mountain View, California aribandi@google.com Dara Bahri Google Research Mountain View, California dbahri@google.com Zhen Qin Google Research Mountain View, California zhenqin@google.com Donald Metzler Google Research Mountain View, California metzler@google.com Abstract In the era of pre-trained language models, Transformers are the de facto choice of model architectures. While recent research has shown promise in entirely convolutional, or CNN, architectures, they have not been ex- plored using the pre-train-fine-tune paradigm. In the context of language models, are con- volutional models competitive to Transform- ers when pre-trained? This paper investigates this research question and presents several in- teresting findings. Across an extensive set of experiments on 8 datasets/tasks, we find that CNN-based pre-trained models are competi- tive and outperform their Transformer counter- part in certain scenarios, albeit with caveats. Overall, the findings outlined in this paper suggest that conflating pre-training and archi- tectural advances is misguided and that both advances should be considered independently. We believe our research paves the way for a healthy amount of optimism in alternative ar- chitectures. 1 Introduction In the modern era of pre-training, there appears to be an unbreakable tie between Transformer ar- chitectures (Vaswani et al., 2017) and pre-trained language models. Models such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and T5 (Raffel et al., 2019) have all adopted Transformers as their underlying architecture. As a matter of fact, there are barely any recent pre-trained models not based on Transformers. While the contextual representation learning has a rich history (Pennington et al., 2014; Dai and Le, ∗Google AI Resident 2015; Chidambaram et al., 2018; Liu et al., 2020; Qiu et al., 2020), modern pre-trained language mod- eling started with models like ELMo (Peters et al., 2018) and CoVE (McCann et al., 2017) which are based on recurrent (e.g. LSTM (Hochreiter and Schmidhuber, 1997)) architectures. Although they were successful, research using these architectures dwindled as Transformers stole the hearts of the NLP community, having, possibly implicitly, been perceived as a unequivocal advancement over its predecessors. Recent work demonstrates the promise of en- tirely convolution-based models (Wu et al., 2019; Gehring et al., 2017) and questions the necessity of self-attentive architectures like Transformers. For example, in (Wu et al., 2019), the proposed convo- lutional seq2seq models outperform Transformers on a series of canonical benchmarks such as ma- chine translation and language modeling. From these findings emerge a rather natural line of ques- tioning - should we consider pre-trained models beyond Transformers? Despite early success, the relevance of convo- lutional models in the era of pre-trained language models remains an open question. To the best of our …

Content truncated to 500 words