Contextual Representation Learning beyond Masked Language Modeling

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2701 - 2714 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Contextual Representation Learning beyond Masked Language Modeling Zhiyi Fu1∗ †, Wangchunshu Zhou2∗, Jingjing Xu3†, Hao Zhou2, Lei Li3† 1Peking University 2ByteDance AI Lab 3University of California, Santa Barbara ypfzy@pku.edu.cn {zhouwangchunshu.7, zhouhao.nlp}@bytedance.com {jingjingxu, leili}@cs.ucsb.edu Abstract How do masked language models (MLMs) such as BERT learn contextual representa- tions? In this work, we analyze the learn- ing dynamics of MLMs. We find that MLMs adopt sampled embeddings as anchors to es- timate and inject contextual semantics to rep- resentations, which limits the efficiency and effectiveness of MLMs. To address these is- sues, we propose TACO, a simple yet effec- tive representation learning approach to di- rectly model global semantics. TACO extracts and aligns contextual semantics hidden in con- textualized representations to encourage mod- els to attend global semantics when gener- ating contextualized representations. Exper- iments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over existing MLMs. The code is available at https:// github.com/FUZHIYI/TACO. 1 Introduction In the age of deep learning, the basis of repre- sentation learning is to learn distributional seman- tics. The target of distributional semantics can be summed up in the so-called distributional hypoth- esis (Harris, 1954): Linguistic items with similar distributions have similar meanings. To model similar meanings, traditional representation ap- proaches (Mikolov et al., 2013; Pennington et al., 2014) (e.g., Word2Vec) model distributional seman- tics by defining tokens using context-independent (CI) dense vectors, i.e., word embeddings, and di- rectly aligning the representations of tokens in the same context. Nowadays, pre-trained language models (PTMs) (Devlin et al., 2019; Radford et al., 2018; Qiu et al., 2020) expand static embeddings into contextualized representations where each to- ken has two kinds of representations: context- independent embedding, and context-dependent ∗Equal Contribution †This work is done at ByteDance AI Lab. Enc. 𝑥! (#) 𝑥% (#) 𝑥& (#) − = Embedding = − = − 𝑥! (') 𝑥% (') 𝑥& (') − = = − = − ( , ) ( , ) Positive Negative ⇒ ⇒ Enc. Last Hidden Embedding Global Anchor from 𝑥(#) or 𝑥(') , , Figure 1: Illustration of the proposed token-alignment contrastive objective. It extracts and aligns the global semantics hidden in contextualized representations via the gap between contextualized representations and cor- responding static embeddings. (CD) dense representation that stems from its em- bedding and contains context information. Al- though language modeling and representation learn- ing have distinct targets, masked language model- ing is still the prime choice to learn token represen- tations with access to large scale of raw texts (Pe- ters et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020). It naturally raises a question: How do masked language models learn contextual representa- tions? Following the widely-accepted understand- ing (Wang and Isola, 2020), MLM optimizes two properties, the alignment of contextualized …

Content truncated to 500 words