Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4050–4060 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4050 Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning Shuai Tang Virginia R. de Sa Department of Cognitive Science, Halıcıo˘glu Data Science Institute, UC San Diego {shuaitang93,desa}@ucsd.edu Abstract Encoder-decoder models for unsupervised sentence representation learning using the dis- tributional hypothesis effectively constrain the learnt representation of a sentence to only that needed to reproduce the next sentence. While the decoder is important to constrain the rep- resentation, these models tend to discard the decoder after training since only the encoder is needed to map the input sentence into a vec- tor representation. However, parameters learnt in the decoder also contain useful information about the language. In order to utilise the de- coder after learning, we present two types of decoding functions whose inverse can be eas- ily derived without expensive inverse calcu- lation. Therefore, the inverse of the decod- ing function can serve as another encoder that produces sentence representations. We show that, with careful design of the decoding func- tions, the model learns good sentence repre- sentations, and the ensemble of the represen- tations produced from the encoder and the in- verse of the decoder demonstrate even better generalisation ability and solid transferability. 1 Introduction Learning sentence representations from unlabelled data is becoming increasingly prevalent in both the machine learning and natural language pro- cessing research communities, as it efficiently and cheaply allows knowledge extraction that can suc- cessfully transfer to downstream tasks. Methods built upon the distributional hypothesis (Harris, 1954) and distributional similarity (Firth, 1957) can be roughly categorised into two types: Word-prediction Objective: This objective pushes the system to make better predictions of words in a given sentence. As the nature of the objective is to predict words, these are also called generative models. In one of the two classes of models of this type, an encoder-decoder model is learnt using a corpus of contiguous sentences (Kiros et al., 2015; Gan et al., 2017; Tang et al., 2018) to make predictions of the words in the next sentence given the words in the current one. Af- ter training, the decoder is usually discarded as it is only needed during training and is not de- signed to produce sentence representations. In the other class of models of this type, a large lan- guage model is learnt (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018) on unlabelled cor- pora, which could be an autoregressive model or a masked language model, which gives extremely powerful language encoders but requires massive computing resources and training time. Similarity-based Objective: The objective here relies on a predefined similarity function to force the model to produce more similar represen- tations for adjacent sentences than those that are not (Li and Hovy, 2014; Jernite et al., 2017; Nie et al., 2017; Logeswaran and Lee, 2018). There- fore, the inductive biases introduced by the two …

Content truncated to 500 words