Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9902–9910 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9902 Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings Marco Di Giovanni Politecnico di Milano Università di Bologna marco.digiovanni@polimi.it Marco Brambilla Politecnico di Milano marco.brambilla@polimi.it Abstract Semantic sentence embeddings are usually su- pervisedly built minimizing distances between pairs of embeddings of sentences labelled as semantically similar by annotators. Since big labelled datasets are rare, in particular for non- English languages, and expensive, recent stud- ies focus on unsupervised approaches that re- quire not-paired input sentences. We instead propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter’s intrinsic powerful signals of relatedness: replies and quotes of tweets. We use the collected pairs to train a Trans- former model with triplet-like structures, and we test the generated embeddings on Twit- ter NLP similarity tasks (PIT and TURL) and STSb. We also introduce four new sentence ranking evaluation benchmarks of informal texts, carefully extracted from the initial col- lections of tweets, proving not only that our best model learns classical Semantic Textual Similarity, but also excels on tasks where pairs of sentences are not exact paraphrases. Abla- tion studies reveal how increasing the corpus size influences positively the results, even at 2M samples, suggesting that bigger collections of Tweets still do not contain redundant infor- mation about semantic similarities.1 1 Introduction and Related Work Word-level embeddings techniques compute fixed- size vectors encoding semantics of words (Mikolov et al., 2013; Pennington et al., 2014), usually unsu- pervisedly trained from large textual corpora. It has always been more challenging to build high-quality sentences-level embeddings. Currently, best sentence-embeddings approaches are supervisedly trained using large labeled datasets (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019; Chen et al., 2019; 1Code available at https://github.com/ marco-digio/Twitter4SSE Du et al., 2021; Wieting et al., 2020; Huang et al., 2021), such as NLI datasets (Bowman et al., 2015; Williams et al., 2018) or paraphrase corpora (Dolan and Brockett, 2005). Round-trip translation has been also exploited, where semantically similar pairs of sentences are generated translating the non- English side of NMT pairs, as in ParaNMT (Wi- eting and Gimpel, 2018) and Opusparcus (Creutz, 2018). However, large labeled datasets are rare and hard to collect, especially for non-English lan- guages, due to the cost of manual labels, and there exist no convincing argument for why datasets from these tasks are preferred over other datasets (Carls- son et al., 2021), even if their effectiveness on STS tasks is largely empirically tested. Therefore, recent works focus on unsupervised approaches (Li et al., 2020; Carlsson et al., 2021; Wang et al., 2021; Giorgi et al., 2020; Logeswaran and Lee, 2018), where unlabeled datasets are ex- ploited to increase the performance of models. These works use classical formal corpora such as OpenWebText (Gokaslan and Cohen, 2019), English Wikipedia, obtained through Wikiextrac- tor …

Content truncated to 500 words