Analogy-Guided Evolutionary Pretraining of Binary Word Embeddings

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 683–693 November 20–23, 2022. ©2022 Association for Computational Linguistics 683 Analogy-Guided Evolutionary Pretraining of Binary Word Embeddings R. Alexander Knipper∗, Md. Mahadi Hassan∗, Mehdi Sadi†, Shubhra Kanti Karmaker Santu∗ BDI Lab, Department of Computer Science & Software Engineering∗ Department of Electrical & Computer Engineering† Auburn University, Alabama, USA {rak0035, mzh0167, mzs0190, sks0086}@auburn.edu Abstract As we begin to see low-powered computing paradigms (Neuromorphic Computing, Spiking Neural Networks, etc.) becoming more popular, learning binary word embeddings has become increasingly important for supporting NLP ap- plications at the edge. Existing binary word embeddings are mostly derived from pretrained real-valued embeddings through different sim- ple transformations, which often break the se- mantic consistency and the so-called “arith- metic” properties learned by the original, real- valued embeddings. This paper aims to address this limitation by introducing a new approach to learn binary embeddings from scratch, preserv- ing the semantic relationships between words as well as the arithmetic properties of the em- beddings themselves. To achieve this, we pro- pose a novel genetic algorithm to learn the re- lationships between words from existing word analogy data-sets, carefully making sure that the arithmetic properties of the relationships are preserved. Evaluating our generated 16, 32, and 64-bit binary word embeddings on Mikolov’s word analogy task shows that more than 95% of the time, the best fit for the anal- ogy is ranked in the top 5 most similar words in terms of cosine similarity. 1 Introduction Word embeddings see very common use in many widely-adopted NLP applications, e.g., document summarization (El-Kassas et al., 2021), sentiment analysis (Yadav and Vishwakarma, 2020), entity extraction (Li et al., 2020), question answering (Jin et al., 2022), etc. However, the majority of commonly-used word embeddings are far too de- manding in terms of energy and computational re- sources required to train and load them, making state-of-the-art word embeddings unsuitable for use in a low-energy environment, like in an internet of things (IoT) device (Zadeh et al., 2020; Wang et al., 2020; Daghero et al., 2021) or in a Neuro- morphic processor (Schuman et al., 2022; Davies et al., 2021). As we observe these low-powered de- vices entering the mainstream, we become increas- ingly aware of our inability to use typical word embeddings in those environments, since typical word embeddings usually require multiple giga- bytes for storage and hundreds (if not thousands) of floating-point multiplications to capture meaning- ful relationships between words. Furthermore, low- energy neuromorphic computers in particular are based on binary “spiking” inputs and perform cal- culations using “accumulation” (sum) operations, therefore not supporting floating-point operations (Poon and Zhou, 2011; Davies et al., 2021). Hence, real-valued embeddings are of little use to these low-energy computing paradigms, which is our main motivation for learning high-quality binary embeddings. An intuitive way to address this issue is to take the pretrained real-valued word embeddings and …

Content truncated to 500 words