Sentence Bottleneck Autoencoders from Transformer Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1822–1831 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1822 Sentence Bottleneck Autoencoders from Transformer Language Models Ivan Montero♣ Nikolaos Pappas♣ Noah A. Smith♣♦ ♣Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen Institute for Artificial Intelligence {ivamon,npappas,nasmith}@cs.washington.edu Abstract Representation learning for text via pretrain- ing a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast to autoencoders, also trained on raw text, but with the objective of learning to encode each input as a vector that allows full reconstruction. Autoencoders are attractive because of their latent space structure and generative proper- ties. We therefore explore the construction of a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer mod- ified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pre- trained transformers on text similarity tasks, style transfer (an example of controlled gener- ation), and single-sentence classification tasks in the GLUE benchmark, while using fewer pa- rameters than large pretrained models.1 1 Introduction Recent research has focused on devising new unsu- pervised pretraining methods from unlabeled data that involves some form of language modeling, pri- marily autoregressive (Peters et al., 2018; Radford et al., 2019), masked (Devlin et al., 2019; Liu et al., 2019; Conneau et al., 2020) and generalized (Rad- ford et al., 2019; Brown et al., 2020; Song et al., 2019), with much success on downstream tasks. Under the hood, most of these methods use trans- formers (Vaswani et al., 2017) for encoding text se- quences, which allows them to learn powerful con- textual word representations that have been used widely for building models in NLP. However, this does not hold for sentence representations derived 1Our code is available at: https://github.com/ ivanmontero/autobot from pretrained transformer language models based on a special token or basic pooling operations. To this end, representation learning methods have been designed to better capture semantic information from pretrained transformer language models, e.g., using Siamese networks trained with a triplet loss (Reimers and Gurevych, 2019) or transforming the desired sentence distribution to a Gaussian distri- bution through normalizing flows (Li et al., 2020). Existing sentence representations directly de- rived from pretrained language models or learned by specialized methods cannot guarantee perfect reconstruction of the input, a property that can en- hance the structure of their semantic space and en- able their use for controlled generation tasks. For the latter, a few recent studies have looked into ways to steer generation of pretrained language models towards a particular style (Dathathri et al., 2020; Krause et al., 2021), although they require following the gradient during the sampling process and rely on style text classifiers which might not be always available. The latent space …

Content truncated to 500 words