A Bilingual Generative Transformer for Semantic Sentence Embedding

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1581–1594, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1581 A Bilingual Generative Transformer for Semantic Sentence Embedding John Wieting1, Graham Neubig1, and Taylor Berg-Kirkpatrick2 1Carnegie Mellon University, Pittsburgh, PA, 15213, USA 2University of California San Diego, San Diego, CA, 92093, USA {jwieting,gneubig}@cs.cmu.edu, tberg@eng.ucsd.edu Abstract Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sen- tences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We pro- pose a deep latent variable model that attempts to perform source separation on parallel sen- tences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vec- tors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilis- tic framework, we introduce priors that encour- age source separation, and can use our model’s posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data gen- erating distributions and inference networks – contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised seman- tic similarity evaluations. Further, we demon- strate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indi- cator of similarity.1 1 Introduction Learning useful representations of language has been a source of recent success in natural language processing (NLP). Much work has been done on learning representations for words (Mikolov et al., 2013; Pennington et al., 2014) and sentences (Kiros et al., 2015; Conneau et al., 2017). More recently, 1Code and data to replicate results available at https: //www.cs.cmu.edu/˜jwieting. deep neural architectures have been used to learn contextualized word embeddings (Peters et al., 2018; Devlin et al., 2018) enabling state-of-the- art results on many tasks. We focus on learning semantic sentence embeddings in this paper, which play an important role in many downstream ap- plications. Since they do not require any labelled data for fine-tuning, sentence embeddings are use- ful out-of-the-box for problems such as measure- ment of Semantic Textual Similarity (STS; Agirre et al. (2012)), mining bitext (Zweigenbaum et al., 2018), and paraphrase identification (Dolan et al., 2004). Semantic similarity measures also have downstream uses such as fine-tuning machine trans- lation systems (Wieting et al., 2019a). There are three main ingredients when design- ing a sentence embedding model: the architecture, the training data, and the objective function. Many architectures including LSTMs (Hill et al., 2016; Conneau et al., 2017; Schwenk and Douze, 2017; Subramanian et al., 2018), Transformers (Cer et al., 2018; Reimers and Gurevych, 2019), and averag- ing models (Wieting et al., 2016b; Arora et al., 2017) are …

Content truncated to 500 words