Better Pre-Training by Reducing Representation Confusion

Findings of the Association for Computational Linguistics: EACL 2023, pages 2325–2336 May 2-6, 2023 ©2023 Association for Computational Linguistics Better Pre-Training by Reducing Representation Confusion Haojie Zhang1, 2∗, Mingfei Liang1∗, Ruobing Xie1∗, Zhenlong Sun1, Bo Zhang1, Leyu Lin1 1WeChat Search Application Department, Tencent, China 2Peking University, China 1 {coldhjzhang, aesopliang, ruobingxie, richardsun, nevinzhang, goshawklin}@tencent.com 2zhanghaojie@stu.pku.edu.cn Abstract In this work, we revisit the Transformer-based pre-trained language models and identify two different types of information confusion in po- sition encoding and model representations, re- spectively. Firstly, we show that in the relative position encoding, the joint modeling about relative distances and directions brings con- fusion between two heterogeneous informa- tion. It may make the model unable to cap- ture the associative semantics of the same dis- tance and the opposite directions, which in turn affects the performance of downstream tasks. Secondly, we notice the BERT with Mask Language Modeling (MLM) pre-training objective outputs similar token representations (last hidden states of different tokens) and head representations (attention weights1of different heads), which may make the diversity of infor- mation expressed by different tokens and heads limited. Motivated by the above investigation, we propose two novel techniques to improve pre-trained language models: Decoupled Di- rectional Relative Position (DDRP) encoding and MTH2 pre-training objective. DDRP de- couples the relative distance features and the directional features in classical relative posi- tion encoding. MTH applies two novel auxil- iary regularizers besides MLM to enlarge the dissimilarities between (a) last hidden states of different tokens, and (b) attention weights of different heads. These designs allow the model to capture different categories of information more clearly, as a way to alleviate information confusion in representation learning for better optimization. Extensive experiments and abla- tion studies on GLUE benchmark demonstrate the effectiveness of our proposed methods. *Equal contribution. 1"attention weights" mainly refer to the dot product be- tween Key and Query in the self-attention module. 2MTH is the abbreviation of our proposed MLM with To- ken Cosine Differentiation (TCD) and Head Cosine Differen- tiation (HCD) pre-training task. TCD and HCD are described in detail in sec. 1(2) and sec.3.2. 1 Introduction The paradigm of pre-training on large-scale cor- pus and fine-tuning on specific task datasets has swept the entire field of Natural Language Pro- cessing (NLP). BERT (Devlin et al., 2018) is the most prominent pre-trained language model, which stacks the encoder blocks of Transformer (Vaswani et al., 2017) and adopts MLM and Next Sentence Prediction (NSP) pre-training tasks, achieving the SOTA results in 2018. After that, a large num- ber of Pre-trained Language Models (PLMs) (Liu et al., 2019; Lan et al., 2020; Raffel et al., 2019; Clark et al., 2020; He et al., 2021) that optimize the Transformer structure and pre-training objec- tives have emerged, which further improves the performance of the pre-trained language models on multiple downstream tasks. In this work, we iden- tify two different types of information confusion in language pre-training, and explore two concep- tually simple and empirically powerful techniques against them as follows: (1) …

Content truncated to 500 words