Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7155–7163 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7155 Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models Keigo Takahashi1, Danushka Bollegala2,3 Tokyo Metropolitan University1, University of Liverpool2, Amazon3 takahashi-keigo@ed.tmu.ac.jp, danushka@liverpool.ac.uk Abstract A variety of contextualised language models have been proposed in the NLP community, which are trained on diverse corpora to produce numerous Neural Language Models (NLMs). However, different NLMs have reported different levels of performances in downstream NLP applications when used as text representations. We propose a sentence-level meta-embedding learning method that takes independently trained contextualised word embedding models and learns a sentence embedding that preserves the complementary strengths of the input source NLMs. Our proposed method is unsupervised and is not tied to a particular downstream task, which makes the learnt meta-embeddings in principle applicable to different tasks that require sentence representations. Specifically, we first project the token-level embeddings obtained by the individual NLMs and learn attention weights that indicate the contributions of source embeddings towards their token-level meta-embeddings. Next, we apply mean and max pooling to produce sentence-level meta-embeddings from token-level meta-embeddings. Experimental results on semantic textual similarity benchmarks show that our proposed unsupervised sentence-level meta-embedding method outperforms previously proposed sentence-level meta-embedding methods as well as a supervised baseline. Keywords: meta-sentence embedding, unsupervised, contextualised language model 1. Introduction Meta-embedding (Yin and Sch¨utze, 2016a; Bollegala et al., 2018) is the task of producing embeddings from a given set of pretrained text embeddings such that the complementary strengths of the input (source) embed- dings are preserved or further enhanced to obtain the best performance on a downstream NLP task. Numer- ous methods have been proposed for creating a meta- embedding from a given set of static word embeddings such as GloVe (Pennington et al., 2014), Skip-gram with Negative Sampling (SGNS) (Mikolov et al., 2013b) or Continuous Bag-of-Words (CBOW) (Mikolov et al., 2013a) as the source embeddings. In contrast to static word embeddings, contextualised word embeddings ob- tained from Neural Language Models (NLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) etc. represent the same word with different vec- tors in different contexts, thereby providing a richer context-dependent semantic representations. Moreover, contextualised word embeddings can be pooled and fine- tuned to obtain accurate sentence-level representations. Compared to static word embeddings, contextualised word/sentence embeddings have reported impressive performances in a wide range of NLP tasks. Challenges: Compared to meta-embedding of static word embeddings, when computing meta-embeddings from contextualised word embeddings (Poerner et al., 2020) one must overcome several challenges. • First, unlike static word embeddings, which rep- resent the same word using the same embedding Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the Univer- sity of Liverpool and is not associated with Amazon. vector in all contexts in which the target word oc- curs, contextualised word embeddings depend on the context of …

Content truncated to 500 words