Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7027–7034 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7027 Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity Nina Poerner∗† and Ulli Waltinger† and Hinrich Sch¨utze∗ ∗Center for Information and Language Processing, LMU Munich, Germany †Corporate Technology Machine Intelligence (MIC-DE), Siemens AG Munich, Germany poerner@cis.uni-muenchen.de | inquiries@cislmu.org Abstract We address the task of unsupervised Seman- tic Textual Similarity (STS) by ensembling di- verse pre-trained sentence encoders into sen- tence meta-embeddings. We apply, extend and evaluate different meta-embedding meth- ods from the word embedding literature at the sentence level, including dimensionality re- duction (Yin and Sch¨utze, 2016), generalized Canonical Correlation Analysis (Rastogi et al., 2015) and cross-view auto-encoders (Bolle- gala and Bao, 2018). Our sentence meta- embeddings set a new unsupervised State of The Art (SoTA) on the STS Benchmark and on the STS12–STS16 datasets, with gains of be- tween 3.7% and 6.4% Pearson’s r over single- source systems. 1 Introduction Word meta-embeddings have been shown to exceed single-source word embeddings on word-level se- mantic benchmarks (Yin and Sch¨utze, 2016; Bolle- gala and Bao, 2018). Presumably, this is because they combine the complementary strengths of their components. There has been recent interest in pre-trained “uni- versal” sentence encoders, i.e., functions that en- code diverse semantic features of sentences into fixed-size vectors (Conneau et al., 2017). Since these sentence encoders differ in terms of their ar- chitecture and training data, we hypothesize that their strengths are also complementary and that they can benefit from meta-embeddings. To test this hypothesis, we adapt different meta- embedding methods from the word embedding lit- erature. These include dimensionality reduction (Yin and Sch¨utze, 2016), cross-view autoencoders (Bollegala and Bao, 2018) and Generalized Canon- ical Correlation Analysis (GCCA) (Rastogi et al., 2015). The latter method was also used by Poerner and Sch¨utze (2019) for domain-specific Duplicate Question Detection. Our sentence encoder ensemble includes three models from the recent literature: Sentence-BERT (Reimers and Gurevych, 2019), the Universal Sen- tence Encoder (Cer et al., 2017) and averaged ParaNMT vectors (Wieting and Gimpel, 2018). Our meta-embeddings outperform every one of their constituent single-source embeddings on STS12–16 (Agirre et al., 2016) and on the STS Benchmark (Cer et al., 2017). Crucially, since our meta-embeddings are agnostic to the contents of their ensemble, future improvements may be possi- ble by adding new encoders. 2 Related work 2.1 Word meta-embeddings Word embeddings are functions that map word types to vectors. They are typically trained on un- labeled corpora and capture word semantics (e.g., Mikolov et al. (2013); Pennington et al. (2014)). Word meta-embeddings combine ensembles of word embeddings by various operations: Yin and Sch¨utze (2016) use concatenation, SVD and lin- ear projection, Coates and Bollegala (2018) show that averaging word embeddings has properties similar to concatenation. Rastogi et al. (2015) apply generalized canonical correlation analysis (GCCA) to an ensemble of word vectors. Bollegala and Bao (2018) learn word meta-embeddings us- ing autoencoder architectures. Neill and Bollegala (2018) evaluate different loss …

Content truncated to 500 words