Word Embedding Binarization with Semantic Information Preservation

Proceedings of the 28th International Conference on Computational Linguistics, pages 1256–1265 Barcelona, Spain (Online), December 8-13, 2020 1256 Word Embedding Binarization with Semantic Information Preservation Samarth Navali *, Praneet Prabhakar Sherki *, Ramesh Inturi, Vanraj Vala Samsung R&D Institute, Bangalore, India (s.pnavali,p.sherki,i.ramesh,vanraj.vala)@samsung.com Abstract With growing applications of Machine Learning in daily lives, Natural Language Processing (NLP) has emerged as a heavily researched area. Finding its applications in tasks ranging from simple Q/A chatbots to fully fledged conversational AI, NLP models are vital. Word and Sen- tence embeddings are one of the most common starting points of any NLP task. A word em- bedding represents a given word in a predefined vector-space while maintaining vector relations with similar or dis-similar entities. As such, different pretrained embedding such as Word2Vec, GloVe, FastText have been developed. These embeddings generated on millions of words are however very large in terms of size. Having embeddings with floating point precision also makes the downstream evaluation slow. In this paper we present a novel method to convert continuous embedding to its binary representation, thus reducing the overall size of the embedding while keeping the semantic and relational knowledge intact. This will facilitate an option of porting such big embedding onto devices where space is limited. We also present different approaches suitable for different downstream tasks based on the requirement of contextual and semantic in- formation. Experiments have shown comparable result in downstream tasks with 7 to 15 times reduction in file size and about 5% change in evaluation parameters. 1 Introduction Natural Language Processing (NLP) is quickly becoming one of the most important branch in Machine Learning field, with companies pouring millions to perfect their NLP engines. NLP is the cornerstone in the road to developing a perfect conversational AI. But NLP also finds its way in more trivial but important tasks in modern life, be it a simple chatbot, a Q/A site, Document Classifier. For most of these tasks, an embedding model or a pre-trained embedding is the perfect starting point. A word embedding is a collection of words, represented as vectors in a predefined space. Within this space the vectors follow all typical vector laws. Similar words are close together, dissimilar words are far off. Vector addition and subtraction are possible. Word Embeddings are very important for downstream tasks like document classification, query response, etc. Though Word Embeddings can be generated to be useful to a specific task on a specific dataset, there are embeddings available that are trained on a very large corpus, making them useful in variety of tasks and in turn making them generic. Word2Vec (Mikolov et al., 2013), Glove (Pennington et al., 2014), FastText (Bojanowski et al., 2016) are some of the best generic embeddings available that are trained on millions of vocabularies. Though training on a large dataset makes the embedding useful, it also makes them very large in size. A 300-dimensional Word2Vec model trained on 3 million datapoints is aroung 3.4 GB. This large size makes the embedding very …

Content truncated to 500 words