Universal Sentence Representation Learning with Conditional Masked Language Model

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6216–6228 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6216 Universal Sentence Representation Learning with Conditional Masked Language Model Ziyi Yang1∗, Yinfei Yang2, Daniel Cer2, Jax Law2, Eric Darve1 1Stanford University {ziyi.yang,darve}@stanford.edu 2Google Research {yinfeiy,cer,jaxlaw}@google.com Abstract This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence repre- sentations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of- the-art performance on SentEval (Conneau and Kiela, 2018), even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previ- ous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics. 1 Introduction Sentence embeddings map sentences into a vector space. The vectors capture rich semantic informa- tion that can be used to measure semantic textual similarity (STS) between sentences or train classi- fiers for a broad range of downstream tasks (Con- neau et al., 2017; Subramanian et al., 2018; Lo- geswaran and Lee, 2018; Cer et al., 2018; Reimers and Gurevych, 2019; Yang et al., 2019a,e). State- of-the-art models are usually trained on supervised tasks such as natural language inference (Conneau et al., 2017), or with semi-structured data like question-answer pairs (Cer et al., 2018) and trans- lation pairs (Subramanian et al., 2018; Yang et al., ∗Work done during internship at Google Research. 2019a). However, labeled and semi-structured data are difficult and expensive to obtain, making it hard to cover many domains and languages. Con- versely, recent efforts to improve language mod- els include the development of masked language model (MLM) pre-training from large scale unla- beled corpora (Devlin et al., 2019; Lan et al., 2020; Liu et al., 2019). While internal MLM model rep- resentations are helpful when fine-tuning on down- stream tasks, they do not directly produce good sen- tence representations, without further supervised (Reimers and Gurevych, 2019) or semi-structured (Feng et al., 2020) fine-tuning. In this paper, we explore an unsupervised ap- proach, called Conditional Masked Language Mod- eling (CMLM), to effectively learn sentence rep- resentations from large scale unlabeled corpora. The CMLM model architecture is illustrated in Fig. 1, which integrates sentence representation learning into MLM training by conditioning on sen- tence level representations produced by adjacent sentences. The model therefore needs to learn ef- fective sentence representations in order to perform good MLM. Since CMLM is fully unsupervised, it can be easily extended to new languages. We explore CMLM for both English and multilingual …

Content truncated to 500 words