Learning Compressed Sentence Representations for On-Device Text Processing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 107–116 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 107 Learning Compressed Sentence Representations for On-Device Text Processing Dinghan Shen1∗, Pengyu Cheng1∗, Dhanasekar Sundararaman1 Xinyuan Zhang1, Qian Yang1, Meng Tang3, Asli Celikyilmaz2, Lawrence Carin1 1 Duke University 2 Microsoft Research 3 Stanford University dinghan.shen@duke.edu Abstract Vector representations of sentences, trained on massive text corpora, are widely used as generic sentence embeddings across a variety of NLP problems. The learned representa- tions are generally assumed to be continuous and real-valued, giving rise to a large memory footprint and slow retrieval speed, which hin- ders their applicability to low-resource (mem- ory and computation) platforms, such as mo- bile devices. In this paper, we propose four different strategies to transform continuous and generic sentence embeddings into a bi- narized form, while preserving their rich se- mantic information. The introduced methods are evaluated across a wide range of down- stream tasks, where the binarized sentence em- beddings are demonstrated to degrade perfor- mance by only about 2% relative to their con- tinuous counterparts, while reducing the stor- age requirement by over 98%. Moreover, with the learned binary representations, the seman- tic relatedness of two sentences can be evalu- ated by simply calculating their Hamming dis- tance, which is more computational efficient compared with the inner product operation be- tween continuous embeddings. Detailed anal- ysis and case study further validate the effec- tiveness of proposed methods. 1 Introduction Learning general-purpose sentence representa- tions from large training corpora has received widespread attention in recent years. The learned sentence embeddings can encapsulate rich prior knowledge of natural language, which has been demonstrated to facilitate a variety of downstream tasks (without fine-tuning the encoder weights). The generic sentence embeddings can be trained either in an unsupervised manner (Kiros et al., 2015; Hill et al., 2016; Jernite et al., 2017; Gan ∗ Equal contribution. et al., 2017; Logeswaran and Lee, 2018; Pagliar- dini et al., 2018), or with supervised tasks such as paraphrase identification (Wieting et al., 2016), natural language inference (Conneau et al., 2017), discourse relation classification (Nie et al., 2017), machine translation (Wieting and Gimpel, 2018), etc. Significant effort has been devoted to design- ing better training objectives for learning sentence embeddings. However, prior methods typically as- sume that the general-purpose sentence represen- tations are continuous and real-valued. However, this assumption is sub-optimal from the following perspectives: i) the sentence embeddings require large storage or memory footprint; ii) it is com- putationally expensive to retrieve semantically- similar sentences, since every sentence represen- tation in the database needs to be compared, and the inner product operation is computationally in- volved. These two disadvantages hinder the appli- cability of generic sentence representations to mo- bile devices, where a relatively tiny memory foot- print and low computational capacity are typically available (Ravi and Kozareva, 2018). In this paper, we aim to mitigate the above is- sues by binarizing the continuous sentence em- …

Content truncated to 500 words