MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 562–574 November 10–11, 2021. ©2021 Association for Computational Linguistics 562 MIRRORWIC: On Eliciting Word-in-Context Representations from Pretrained Language Models Qianchu Liu∗, Fangyu Liu∗, Nigel Collier, Anna Korhonen, Ivan Vuli´c Language Technology Lab, TAL, University of Cambridge {ql261, fl399, nhc30, alk23, iv250}@cam.ac.uk Abstract Recent work indicated that pretrained lan- guage models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via sim- ple self-supervised techniques. Inspired by this line of work, in this paper we pro- pose a fully unsupervised approach to improv- ing word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MIRROR- WIC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context- aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MIRROR- WIC models obtain substantial gains over off- the-shelf PLMs across all monolingual, mul- tilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MIRROR- WIC is even on-par with supervised models fine-tuned with in-task data and sense labels. 1 Introduction Pretrained Language Models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) provide dynamic contextual represen- tations; they induce token-level lexical representa- tions that capture the impact of the word’s context on its embedding. Recent studies have assessed the PLMs by probing into their off-the-shelf represen- tation/feature space (Garí Soler et al., 2019; Wiede- mann et al., 2019; Reif et al., 2019; Garí Soler and Apidianaki, 2021). While off-the-shelf PLMs al- ready offer a useful contextualised lexical semantic space, their contextualised representation spaces suffer from instability and anisotropy (Mickus et al., ∗Equal contribution. Due to the fat-tailed nature of pandemic risk… Due t_fat-tailed nature of _mic risk… A review article in nature reported this finding. The end doesn justify the m j nature nature nature Due to the fat-tailed nature of pandemic risk… Due t[MASK] fat-tailed nature of [MASK]mic risk… f( ) f( ) f( ) f( ) random span deletion …… push pull Human interaction with nature has played a major role. The end doesn’t always justify the means. randomly sampled raw sentences Figure 1: An illustrative overview of the MIRROR- WIC method, based on contrastive learning, for elicit- ing better word-in-context (WiC) representations from pretrained language models. We augment a randomly selected WiC instance with random span masking and apply dropout to the hidden states to create two slightly different representations of the base instance. These two representations form a positive pair for contrastive fine-tuning. During fine-tuning, we pull the representa- tions of each positive pair closer together, while at the same time pushing away representations of other WiC instances, serving as negative examples. 2020; Pedinotti and Lenci, 2020). As a conse- quence, they usually fall far behind the perfor- mance of the same PLM fine-tuned with (i) sense annotations …

Content truncated to 500 words