BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2103–2115 August 1–6, 2021. ©2021 Association for Computational Linguistics 2103 BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks Jong-Hoon Oh§ Ryu Iida§¶ Julien Kloetzer§ Kentaro Torisawa§¶ Data-driven Intelligent System Research Center (DIRECT), National Institute of Information and Communications Technology (NICT)§ Graduate School of Science and Technology, NAIST¶ {rovellia, ryu.iida, julien, torisawa}@nict.go.jp Abstract Transformer-based language models (TLMs), such as BERT, ALBERT and GPT-3, have shown strong performance in a wide range of NLP tasks and currently dominate the field of NLP. However, many researchers wonder whether these models can maintain their dom- inance forever. Of course, we do not have answers now, but, as an attempt to find bet- ter neural architectures and training schemes, we pretrain a simple CNN using a GAN-style learning scheme and Wikipedia data, and then integrate it with standard TLMs. We show that on the GLUE tasks, the combination of our pretrained CNN with ALBERT outper- forms the original ALBERT and achieves a similar performance to that of SOTA. Fur- thermore, on open-domain QA (Quasar-T and SearchQA), the combination of the CNN with ALBERT or RoBERTa achieved stronger per- formance than SOTA and the original TLMs. We hope that this work provides a hint for developing a novel strong network architec- ture along with its training scheme. Our source code and models are available at https://github.com/nict-wisdom/bertac. 1 Introduction Transformer-based language models (TLMs) such as BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and GPT-3 (Brown et al., 2020) have shown that large-scale self-supervised pretraining leads to strong performance on various NLP tasks. Many researchers have used TLMs for various downstream tasks, possibly as subcomponents of their methods, and/or they have focused on scaling up TLMs or improving their pretraining schemes. As a result, other architectures like Recurrent Neu- ral Networks (RNN) (Hochreiter and Schmidhu- ber, 1997; Cho et al., 2014) and Convolutional Neural Networks (CNN) (LeCun et al., 1999) are fading away. In this work, we propose a method !! !" !# !!!"" !#"" !"" !"" !!! #$%&'( ... !!! !"#$"#!"#$%&'()'*+ !!! !"#$$%&%'( )*+,-.&/(0-" )*+,-.&/(0-1 )*+,-.&/(0-2 !"# #$%&'( %&$"#!"!"#$"#%"!#$!"#$"#%"" )!*+,-!"#$"#%"! )+./,-!"#$"#%"" )+./, !#!###" 01'(#1%2'"3-456#1'6- 78(6-'9:'66%;<$- +34(%%5678 !"" !8;2'(1%;<-18-';1%13= 9#$>'6-$';1';?'$ !!!$" !#$" !!!%" !#%" Figure 1: Overall architecture of BERTAC under the setting of classification for a two-sentence input (sentencex and sentencey). for improving TLMs by integrating a simple con- ventional CNN to them. We pretrained this CNN on Wikipedia using a Generative Adversarial Net- work (GAN) style training scheme (Goodfellow et al., 2014), and then combined it with TLMs. Oh et al. (2019) similarly used GAN-style train- ing to improve a QA model using a CNN, but their training scheme was applicable only to QA- specific datasets. On the other hand, similarly to TLM, our proposed method for training the CNN is independent of specific tasks. We show that the combination of this CNN with TLMs can achieve …

Content truncated to 500 words