Sentence Analogies: Linguistic Regularities in Sentence Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 3389–3400 Barcelona, Spain (Online), December 8-13, 2020 3389 Sentence Analogies: Linguistic Regularities in Sentence Embeddings Xunjie Zhu ∗ ByteDance zhuxunjie@bytedance.com Gerard de Melo Hasso Plattner Institute, University of Potsdam gdm@demelo.org Abstract While important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. Word vectors are often evaluated by assessing to what degree they exhibit regularities with regard to relationships of the sort considered in word analogies. In this paper, we investigate to what extent commonly used sentence vector representation spaces as well reflect certain kinds of regularities. We propose a number of schemes to induce evaluation data, based on lexical analogy data as well as semantic relationships between sentences. Our experiments consider a wide range of sentence embedding methods, including ones based on BERT-style contextual embeddings. We find that different models differ substantially in their ability to reflect such regularities. 1 Introduction Sentence embeddings are dense vectors that reflect salient semantic properties of a sentence. Similar to how commonly used word embedding methods such as word2vec (Mikolov et al., 2013a) capture semantic relationships between words, sentence embeddings are expected to encode semantic relationships between sentences. A number of different sentence embedding methods have been proposed (cf. Section 2.1 for an overview). In recent years, pretrained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) have become the method of choice when encoding text. Thus, such models are also often invoked to represent sentences by means of individual embeddings.1 While important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. A particularly prominent aspect of word vector representations induced by methods such as word2vec is that the vector space exhibits certain kinds of regularities. Many of these are of the sort considered in word analogies. Proportional analogies take the form A is to B as C is to D, e.g., Paris is to France as Berlin is to Germany. Rumelhart and Abrahamson (1973) first proposed identifying such analogies using vector representations in a Euclidean space. Given vector representations of concepts, derived from human similarity judgments using a multi-dimensional scaling algorithm, they proposed representing analogical relationships in terms of the difference vectors. Turney and Littman (2005) investigated identifying such analogies using bag-of-words vector space models. Mikolov et al. (2013b) showed that word2vec’s word vector representations reflect certain kinds of word analogies surprisingly well. The widely used word analogy task that they proposed takes the following form. Given embeddings ⃗vA, ⃗vB, ⃗vC, ⃗vD for words A, B, C, D for an analogy of the above form, the task consists in identifying the correct word D given A, B, and C. Most commonly, this is This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. ∗The work was done when …

Content truncated to 500 words