Towards Lossless Encoding of Sentences

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1577–1583 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1577 Towards Lossless Encoding of Sentences Gabriele Prato∗ Mathieu Duchesneau Sarath Chandar Alain Tapp Mila, Universit´e de Montr´eal Abstract A lot of work has been done in the field of im- age compression via machine learning, but not much attention has been given to the compres- sion of natural language. Compressing text into lossless representations while making fea- tures easily retrievable is not a trivial task, yet has huge benefits. Most methods designed to produce feature rich sentence embeddings fo- cus solely on performing well on downstream tasks and are unable to properly reconstruct the original sequence from the learned embed- ding. In this work, we propose a near lossless method for encoding long sequences of texts as well as all of their sub-sequences into fea- ture rich representations. We test our method on sentiment analysis and show good perfor- mance across all sub-sentence and sentence embeddings. 1 Introduction Compressing information by encoding it into a fixed size representation in such a way that per- fect decoding is possible is challenging. Instead, most of the existing sentence encoding methods focus more on learning encoding such that the encoded representations are good enough for the downstream tasks. In this work, we focus on per- fectly decodable encoding of sentences which will be very useful in designing good generative mod- els that can generate longer sentences. Early efforts such as (Hinton and Salakhutdi- nov, 2006) have shown autoencoders to effectively yield compressed input representations. Pollack (1990) was the first to propose using autoencoders recursively. Such models have been shown to be useful for a multitude of tasks. Luong et al. (2013) use recursive neural networks and neural language models to better represent rare words ∗Corresponding author: gabriele.prato@umontreal.ca via morphemes. Socher et al. (2011a) use recur- sive autoencoders for paraphrase detection, learn- ing sentence embeddings (Socher et al., 2010) and syntactic parsing. Socher et al. (2011b) also use a recursive autoencoder to build a tree struc- ture based on error reconstruction. Additionally, Socher et al. (2012) use a matrix-vector RNN to learn semantic relationships present in natural lan- guage and show good performance on such task as well as sentiment classification. Then, Socher et al. (2013) introduced the Recursive Neural Ten- sor Network, trained on a their proposed Senti- ment Treebank corpus to better deal with negat- ing sub-sequences for better sentiment classifica- tion. Recently, Kokkinos and Potamianos (2017) proposed Structural Attention to build syntactic trees and improve even further performance on SST. Parse trees do alleviate the burden of learn- ing the syntactic structure of text, but these meth- ods limit the number of generated embeddings to the number of nodes in the parse tree. Our pro- posed method does not have such a restriction as all possible syntactic tree can be simultaneously represented by the architecture. Convolutional Neural Networks (LeCun et al., 1989) have …

Content truncated to 500 words