Transferable Neural Projection Representations

Proceedings of NAACL-HLT 2019, pages 3355–3360 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3355 Transferable Neural Projection Representations Chinnadhurai Sankar Mila, Universit´e de Montr´eal∗ Montreal, QC, Canada chinnadhurai@gmail.com Sujith Ravi Google Research Mountain View, CA, USA sravi@google.com Zornitsa Kozareva Google Mountain View, CA, USA zornitsa@kozareva.com Abstract Neural word representations are at the core of many state-of-the-art natural language process- ing models. A widely used approach is to pre- train, store and look up word or character em- bedding matrices. While useful, such represen- tations occupy huge memory making it hard to deploy on-device and often do not generalize to unknown words due to vocabulary pruning. In this paper, we propose a skip-gram based architecture coupled with Locality-Sensitive Hashing (LSH) projections to learn efficient dynamically computable representations. Our model does not need to store lookup tables as representations are computed on-the-fly and require low memory footprint. The representa- tions can be trained in an unsupervised fash- ion and can be easily transferred to other NLP tasks. For qualitative evaluation, we analyze the nearest neighbors of the word representa- tions and discover semantically similar words even with misspellings. For quantitative evalu- ation, we plug our transferable projections into a simple LSTM and run it on multiple NLP tasks and show how our transferable projec- tions achieve better performance compared to prior work. 1 Introduction Pre-trained word representations are at the core of many neural language understanding models. Among the most popular and widely used word embeddings are word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) and ELMO (Peters et al., 2018). The biggest challenge with word embedding is that they require lookup and a large memory footprint, as we have to store one entry (d-dim vector) per word and it blows up. In parallel, the tremendous success of deep learn- ing models and the explosion of mobile, IoT de- ∗Work done during internship at Google. vices coupled together with the growing user pri- vacy concerns have led to the need for deploying deep learning models on-device for inference. This has led to new research in compressing large and complex deep learning models for low power on- device deployment. Recently, (Ravi and Kozareva, 2018) developed an on-device neural text classifi- cation model. They proposed to reduce the mem- ory footprint of large neural networks by replacing the input word embeddings with projection based representations. (Ravi and Kozareva, 2018) used n-gram features to generate binary LSH (Charikar, 2002) randomized projections on the fly surpassing the need to store word emebdding tables and reduc- ing the memory size. The projection models reduce the memory occupied by the model from O(|V |) to O(nP), where |V | refers to the vocabulary size and nP refers to number of projection operations (Ravi, 2017). Two key advantages of the projection based representations over word embeddings are: (1) they are fixed and have low memory size; (2) they can handle out of vocabulary words. However, the projections in (Ravi and Kozareva, …

Content truncated to 500 words