On-Device Text Representations Robust To Misspellings via Projections

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2871–2876 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2871 On-Device Text Representations Robust To Misspellings via Projections Chinnadhurai Sankar Mila, Universit´e de Montr´eal∗ Montreal, QC, Canada chinnadhurai@gmail.com Sujith Ravi Amazon Alexa† Sunnyvale, CA, USA sravi@sravi.org Zornitsa Kozareva Google Mountain View, CA, USA zornitsa@kozareva.com Abstract Recently, there has been a strong interest in developing natural language applications that live on personal devices such as mobile phones, watches and IoT with the objective to preserve user privacy and have low mem- ory. Advances in Locality-Sensitive Hashing (LSH)-based projection networks have demon- strated state-of-the-art performance in various classification tasks without explicit word (or word-piece) embedding lookup tables by com- puting on-the-fly text representations. In this paper, we show that the projection based neural classifiers are inherently robust to misspellings and perturbations of the input text. We empirically demonstrate that the LSH projection based classifiers are more robust to common misspellings compared to BiL- STMs (with both word-piece & word-only to- kenization) and fine-tuned BERT based meth- ods. When subject to misspelling attacks, LSH projection based classifiers had a small av- erage accuracy drop of 2.94% across multi- ple classifications tasks, while the fine-tuned BERT model accuracy had a significant drop of 11.44%. 1 Introduction At the core of Natural Language Processing (NLP) neural models are pre-trained word embeddings like Word2Vec (Mikolov et al., 2013), GloVe (Pen- nington et al., 2014a) and ELMo (Peters et al., 2018). They help initialize the neural models, lead to faster convergence and have improved per- formance for numerous application such as Ques- tion Answering (Liu et al., 2018), Summarization (Cheng and Lapata, 2016), Sentiment Analysis (Yu et al., 2017). While word embeddings are powerful in unlimited constraints such as computation power ∗Work done during internship at Google †Work done while at Google AI and compute resources, it becomes challenging to deploy them to on-device due to their huge size. Figure 1: One-hot word embedding look-up vectors vs linear combination of LSH projection based vectors (Ravi, 2017) representing the same word. This led to interesting research by (Ravi and Kozareva, 2018; Sankar et al., 2019), who showed that word embeddings can be replaced with lightweight binary Locality-Sensitive Hashing (LSH) based projections learned on-the-fly. The projection approach surmounts the need to store any embedding matrices, since the projections are dynamically computed. This further enables user privacy by performing inference directly on device without sending user data (e.g., personal informa- tion) to the server. The embedding memory size is reduced from O(V ) to O(K), where V is the token vocabulary size and K << V , is the binary LSH projection size. The projection representa- tions can operate on either word or character level, and can be used to represent a sentence or a word depending on the NLP application. For instance, recently the Projection Sequence Networks (ProS- eqo) (Kozareva and Ravi, 2019) used BiLSTMs over word-level projection representations to repre- …

Content truncated to 500 words