Modeling the Influence of Verb Aspect on the Activation of Typical Event Locations with BERT

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 13–27 August 1–6, 2021. ©2021 Association for Computational Linguistics 13 LV-BERT: Exploiting Layer Variety for BERT Weihao Yu National University of Singapore weihaoyu6@gmail.com Zihang Jiang National University of Singapore jzihang@u.nus.edu Fei Chen Huawei Noah’s Ark Lab chen.f@huawei.com Qibin Hou National University of Singapore andrewhoux@gmail.com Jiashi Feng National University of Singapore elefjia@nus.edu.sg Abstract Modern pre-trained language models are mostly built upon backbones stacking self- attention and feed-forward layers in an in- terleaved order. In this paper, beyond this stereotyped layer pattern, we aim to improve pre-trained models by exploiting layer vari- ety from two aspects: the layer type set and the layer order. Specifically, besides the origi- nal self-attention and feed-forward layers, we introduce convolution into the layer type set, which is experimentally found beneficial to pre-trained models. Furthermore, beyond the original interleaved order, we explore more layer orders to discover more powerful archi- tectures. However, the introduced layer variety leads to a large architecture space of more than billions of candidates, while training a single candidate model from scratch already requires huge computation cost, making it not afford- able to search such a space by directly training large amounts of candidate models. To solve this problem, we first pre-train a supernet from which the weights of all candidate models can be inherited, and then adopt an evolutionary algorithm guided by pre-training accuracy to find the optimal architecture. Extensive exper- iments show that LV-BERT model obtained by our method outperforms BERT and its variants on various downstream tasks. For example, LV-BERT-small achieves 78.8 on the GLUE testing set, 1.8 higher than the strong baseline ELECTRA-small. 1 1 Introduction In recent years, pre-trained language models, such as the representative BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020), have gained great success in natural language processing tasks (Pe- ters et al., 2018a; Radford et al., 2018; Yang et al., 2019; Clark et al., 2020). The backbone architec- tures of these models mostly adopt a stereotyped 1https://github.com/yuweihao/LV-BERT Layer Variety Layer Types Layer Orders Self-Attention Feed-Forward Convolution Interleaved Sandwich Random Searched ① ② ③ ④ ⑤ ⑥ ⑦ (a) {①②} × ④ → BERT/ELECTRA {②③} × ④ → DynamicConv {①②} × ⑤ → Sandwich {①②③} × ⑦ → LV-BERT (b) 60 65 70 75 80 85 GLUE average acuracy on dev set LV-BERT Sandwich DynamicConv ELECTRA BERT 81.8 78.6 64.4 80.4 75.1 (c) Figure 1: (a) Illustration of layer variety. This concept consists of two aspects: layer type and layer order. (b) Different models represented by layer variety. (c) Per- formance of different models with hidden size of 256 on GLUE (Wang et al., 2018) development set. Except BERT pre-trained with the Masked Language Model- ing objective (Devlin et al., 2019), the other models are pre-trained with Replaced Token Detection objec- tive (Clark et al., 2020) to save computation cost. layer pattern, in which the self-attention and feed- forward layers are arrayed in an interleaved order (Vaswani et al., 2017). However, there …

Content truncated to 500 words