Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442–1459 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1442 Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders Fangyu Liu, Ivan Vuli´c, Anna Korhonen, Nigel Collier Language Technology Lab, TAL, University of Cambridge {fl399, iv250, alk23, nhc30}cam.ac.uk Abstract Previous work has indicated that pretrained Masked Language Models (MLMs) are not ef- fective as universal lexical and sentence en- coders off-the-shelf, i.e., without further task- specific fine-tuning on NLI, sentence similar- ity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective lexical and sentence encoders even without any addi- tional data, relying simply on self-supervision. We propose an extremely simple, fast, and ef- fective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20–30 seconds with no access to additional external knowledge. Mirror-BERT relies on identical and slightly modified string pairs as positive (i.e., synonymous) fine-tuning exam- ples, and aims to maximise their similarity dur- ing “identity fine-tuning”. We report huge gains over off-the-shelf MLMs with Mirror- BERT both in lexical-level and in sentence- level tasks, across different domains and differ- ent languages. Notably, in sentence similarity (STS) and question-answer entailment (QNLI) tasks, our self-supervised Mirror-BERT model even matches the performance of the Sentence- BERT models from prior work which rely on annotated task data. Finally, we delve deeper into the inner workings of MLMs, and sug- gest some evidence on why this simple Mirror- BERT fine-tuning approach can yield effective universal lexical and sentence encoders. 1 Introduction Transfer learning with pretrained Masked Lan- guage Models (MLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) has been widely successful in NLP, offering unmatched per- formance in a large number of tasks (Wang et al., 2019a). Despite the wealth of semantic knowledge stored in the MLMs (Rogers et al., 2020), they do not produce high-quality lexical and sentence em- beddings when used off-the-shelf, without further dist( , ) < dist( / , / /…) f(x1) f(¯x1) f(x1) f(¯x1) f(x2) f(x3) BERT/ RoBERTa BERT/ RoBERTa weight sharing ( random span masking ) multiple dropout layers x1 ¯x1 f(x1) f(¯x1) f(x2) f(x3) f(x4) f(x5) Figure 1: Illustration of the main concepts behind the proposed self-supervised Mirror-BERT method. The same text sequence can be observed from two addi- tional “views”: 1) by performing random span mask- ing in the input space, and/or 2) by applying dropout (inside the BERT/RoBERTa MLM) in the feature space, yielding identity-based (i.e., “mirrored”) positive exam- ples for fine-tuning. A contrastive learning objective is then applied to encourage such “mirrored” positive pairs to obtain more similar representations in the em- bedding space relatively to negative pairs. task-specific fine-tuning (Feng et al., 2020; Li et al., 2020). In fact, previous work has shown that their performance is sometimes even below static word embeddings and specialised sentence encoders (Cer et …

Content truncated to 500 words