ProFormer: Towards On-Device LSH Projection Based Transformers

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2823–2828 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2823 ProFormer: Towards On-Device LSH Projection Based Transformers Chinnadhurai Sankar Mila, Universit´e de Montr´eal∗ Montreal, QC, Canada chinnadhurai@gmail.com Sujith Ravi Amazon Alexa† Sunnyvale, CA, USA sravi@sravi.org Zornitsa Kozareva Google Mountain View, CA, USA zornitsa@kozareva.com Abstract At the heart of text based neural models lay word representations, which are powerful but occupy a lot of memory making it challeng- ing to deploy to devices with memory con- straints such as mobile phones, watches and IoT. To surmount these challenges, we intro- duce ProFormer – a projection based trans- former architecture that is faster and lighter making it suitable to deploy to memory con- straint devices and preserve user privacy. We use LSH projection layer to dynamically gen- erate word representations on-the-fly without embedding lookup tables leading to significant memory footprint reduction from O(V.d) to O(T), where V is the vocabulary size, d is the embedding dimension size and T is the dimen- sion of the LSH projection representation. We also propose a local projection attention (LPA) layer, which uses self-attention to transform the input sequence of N LSH word projections into a sequence of N/K representations reduc- ing the computations quadratically by O(K2). We evaluate ProFormer on multiple text classi- fication tasks and observed improvements over prior state-of-the-art on-device approaches for short text classification and comparable perfor- mance for long text classification tasks. Pro- Former is also competitive with other popu- lar but highly resource-intensive approaches like BERT and even outperforms small-sized BERT variants with significant resource sav- ings – reduces the embedding memory foot- print from 92.16 MB to 1.7 KB and requires 16× less computation overhead, which is very impressive making it the fastest and smallest on-device model. 1 Introduction Transformers (Vaswani et al., 2017) based archi- tectures like BERT (Devlin et al., 2018), XL-net ∗Work done during internship at Google †Work done while at Google AI (Yang et al., 2019), GPT-2 (Radford et al., 2019), MT-DNN (Liu et al., 2019a), RoBERTA (Liu et al., 2019b) reached state-of-the-art performance on tasks like machine translation (Arivazhagan et al., 2019), language modelling (Radford et al., 2019), text classification benchmarks like GLUE (Wang et al., 2018). However, these models require huge amount of memory and need high computational requirements making it hard to deploy to small memory constraint devices such as mobile phones, watches and IoT. Recently, there have been in- terests in making BERT lighter and faster (Sanh et al., 2019; McCarley, 2019). In parallel, recent on-device works like SGNN (Ravi and Kozareva, 2018), SGNN++ (Ravi and Kozareva, 2019) and (Sankar et al., 2019) produce lightweight models with extremely low memory footprint. They em- ploy a modified form of LSH projection to dynam- ically generate a fixed binary projection represen- tation, P(x) ∈ [0, 1]T for the input text x using word or character n-grams and skip-grams features, and a 2-layer MLP + softmax …

Content truncated to 500 words