EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3870 - 3885 July 10-15, 2022 ©2022 Association for Computational Linguistics EASE: Entity-Aware Contrastive Learning of Sentence Embedding Sosuke Nishikawa†,‡∗ sosuke-nishikawa@alumni.u-tokyo.ac.jp Ryokan Ri†,‡ ryo0123@ousia.jp Ikuya Yamada†,§ ikuya@ousia.jp Yoshimasa Tsuruoka‡ tsuruoka@logos.t.u-tokyo.ac.jp Isao Echizen‡,¶ iechizen@nii.ac.jp †Studio Ousia ‡The University of Tokyo §RIKEN AIP ¶National Institute of Informatics Abstract We present EASE, a novel method for learn- ing sentence embeddings via contrastive learn- ing between sentences and their related entities. The advantage of using entity supervision is twofold: (1) entities have been shown to be a strong indicator of text semantics and thus should provide rich training signals for sen- tence embeddings; (2) entities are defined in- dependently of languages and thus offer useful cross-lingual alignment supervision. We eval- uate EASE against other unsupervised mod- els both in monolingual and multilingual set- tings. We show that EASE exhibits competi- tive or better performance in English seman- tic textual similarity (STS) and short text clus- tering (STC) tasks and it significantly outper- forms baseline methods in multilingual settings on a variety of tasks. Our source code, pre- trained models, and newly constructed multi- lingual STC dataset are available at https: //github.com/studio-ousia/ease. 1 Introduction The current dominant approach to learning sen- tence embeddings is fine-tuning general-purpose pretrained language models, such as BERT (Devlin et al., 2019), with a particular training supervision. The type of supervision can be natural language inference data (Reimers and Gurevych, 2019), ad- jacent sentences (Yang et al., 2021), or a parallel corpus for multilingual models (Feng et al., 2020). In this paper, we explore a type of supervision that has been under-explored in the literature: entity hyperlink annotations from Wikipedia. Their ad- vantage is twofold: (1) entities have been shown to be a strong indicator of text semantics (Gabrilovich and Markovitch, 2007; Yamada et al., 2017, 2018; Ling et al., 2020) and thus should provide rich training signals for sentence embeddings; (2) en- tities are defined independently of languages and ∗ Work done as an intern at Studio Ousia. Figure 1: Illustration of the main concept behind EASE. Using a contrastive framework, sentences are embedded in the neighborhood of their hyperlink entity embed- dings and kept apart from irrelevant entities. Here, we share the entity embeddings across languages for multi- lingual models to facilitate cross-lingual alignment of the representation. thus offer a useful cross-lingual alignment supervi- sion (Calixto et al., 2021; Nishikawa et al., 2021; Jian et al., 2022; Ri et al., 2022). The extensive multilingual support of Wikipedia alleviates the need for a parallel resource to train well-aligned multilingual sentence embeddings, especially for low-resource languages. To demonstrate the effec- tiveness of entity-based supervision, we present EASE (Entity-Aware contrastive learning of Sen- tence Embeddings), which produces high-quality sentence embeddings in both monolingual and mul- tilingual settings. EASE learns sentence embeddings with two types of objectives: (1) our novel entity contrastive learning (CL) loss between sentences and their re- lated entities (Figure …

Content truncated to 500 words