The Chinese Remainder Theorem for Compact, Task-Precise, Efficient and Secure Word Embeddings

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3512–3521 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3512 The Chinese Remainder Theorem for Compact, Task-Precise, Efficient and Secure Word Embeddings Patricia Thaine Department of Computer Science University of Toronto pthaine@cs.toronto.edu Gerald Penn Department of Computer Science University of Toronto gpenn@cs.toronto.edu Abstract The growing availability of powerful mobile devices and other edge devices, together with increasing regulatory and security concerns about the exchange of personal information across networks of these devices has chal- lenged the Computational Linguistics commu- nity to develop methods that are at once fast, space-efficient, accurate and amenable to se- cure encoding schemes such as homomorphic encryption. Inspired by recent work that re- stricts floating point precision to speed up neu- ral network training in hardware-based SIMD, we have developed a method for compressing word vector embeddings into integers using the Chinese Reminder Theorem that speeds up addition by up to 48.27% and at the same time compresses GloVe word embedding libraries by up to 25.86%. We explore the practical- ity of this simple approach by investigating the trade-off between precision and performance in two NLP tasks: compositional semantic re- latedness and opinion target sentiment classi- fication. We find that in both tasks, lowering floating point number precision results in neg- ligible changes to performance. 1 Introduction In recent years, NLP models, particularly lan- guage models, have come under increasing scrutiny for their potential privacy leaks (e.g., Car- lini et al. (2018)). One answer has been to push NLP models onto edge devices, such as mobile phones and browsers, for on-device inference us- ing differentially private federated learning (e.g., Yang et al. (2018)). Edge computing, in turn, re- quires smaller and faster models, such as Distill- BERT (Sanh et al., 2019) or mobileBERT (Sun et al., 2020), which theoretically improves the vi- ability of using BERT embeddings on these de- vices. What has yet to take place, however, is careful numerical analysis of word embeddings at vary- ing precisions for different NLP tasks. Numerical analysis can inform potential alternatives to defin- ing the representations of word embeddings them- selves. We must carefully orchestrate a balance among the size of these representations, their se- curity, the algebra of operations that they enable, and, because many edge devices are resource- restricted relative to the GPU clusters that we do research on, the efficiency and power consumption of those operations. In particular, feature/basic phones are growing in popularity in developing nations12, often with the possibility of expand- ing their memory to about 16GB with a MicroSD. Many privacy-enabled language models, in partic- ular, are simply out of reach for these low-resource devices. No work to date has proposed using the Chinese Remainder Theorem (CRT) to create software- based compressed word embeddings, adapted to Single Instruction Multiple Data (SIMD). We pro- pose here a CRT-based method which speeds up addition by up to 48.27% and compresses GloVe (Pennington et al., 2014) word …

Content truncated to 500 words