Vec2Sent: Probing Sentence Embeddings with Natural Language Generation

Proceedings of the 28th International Conference on Computational Linguistics, pages 1729–1736 Barcelona, Spain (Online), December 8-13, 2020 1729 Vec2Sent: Probing Sentence Embeddings with Natural Language Generation Martin Kerscher and Steffen Eger Natural Language Learning Group (NLLG) Technische Universit¨at Darmstadt martin.kerscher@protonmail.com, eger@aiphes.tu-darmstadt.de Abstract We unveil the language encoded in sentence embeddings by conditionally generating from them. We perceive of this as a new unsupervised probing task and show that it correlates well with downstream task performance. We also illustrate how the language generated from different encoders differs. We apply our approach to generate sentence analogies from sentence embeddings. 1 Introduction Generalizing the concept of word embeddings to sentence level, sentence embeddings (a.k.a. sentence encoders) are ubiquitous in NLP as features in downstream classification tasks and in semantic similarity and retrieval applications (Kiros et al., 2015; Conneau et al., 2017). Probing sentence encoders for the linguistic information signals they contain has likewise become an important field of research, as this allows to introspect otherwise black-box representations (Adi et al., 2017; Conneau et al., 2018). The idea behind probing tasks is to query representations for certain kinds of linguistic information such as the dependency tree depth of an encoded sentence. There are a variety of problems surrounding current probing task specifications: (i) probing tasks need to be manually construed, which brings with it a certain degree of arbitrariness and incompleteness; (ii) most probing tasks require labeled datasets or trained classifiers such as dependency parsers for linguistic processing—however, these may be unavailable for many low-resource languages or available only to a limited degree; (iii) it is not entirely clear how probing tasks have to be designed, e.g., how much training data they require and which classifier to use for probing (Eger et al., 2020); (iv) Ravichander et al. (2020) also argue that standard probing tasks do not outline the information signals a classifier actually uses for making predictions. Our contribution is to design an alternative, more direct introspection of sentence embeddings, namely, through conditional natural language generation, which we call “vec2sent” (V2S). By retrieving and (manually) investigating the discrete output obtained from a dense vector representation, linguistic properties of the embedding may be ‘directly’ unveiled: e.g., we expect that a word-order insensitive model would have a comparatively hard time in restoring the correct syntax of an encoded sentence. V2S requires no labeled data, making it applicable to any language that has at least several ten thousands of written sentences available—e.g., it is particularly suitable for multilingual probing (Krasnowska-Kiera´s and Wr´oblewska, 2019; Eger et al., 2020). Since V2S makes the opaque space Rd observable, it may also reveal intriguing properties of how encoders encode text (cf. Table 2), without having to ‘guess’ relevant probing tasks.1 2 Approach For an input sentence ⃗x ∈ V ∗ (a sequence of tokens over a vocabulary V ), consider its sentence representation x ∈ Rd induced by some sentence embedding model E. We consider a decoder D which takes x as input and produces natural language text as output. …

Content truncated to 500 words