HashFormers: Towards Vocabulary-independent Pre-trained Transformers

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7862 - 7874 December 7-11, 2022 ©2022 Association for Computational Linguistics HashFormers: Towards Vocabulary-independent Pre-trained Transformers Huiyin Xue and Nikolaos Aletras Department of Computer Science, University of Sheffield United Kingdom {hxue12, n.aletras}@sheffield.ac.uk Abstract Transformer-based pre-trained language mod- els are vocabulary-dependent, mapping by de- fault each token to its corresponding embed- ding. This one-to-one mapping results into em- bedding matrices that occupy a lot of memory (i.e. millions of parameters) and grow linearly with the size of the vocabulary. Previous work on on-device transformers dynamically gener- ate token embeddings on-the-fly without em- bedding matrices using locality-sensitive hash- ing over morphological information. These embeddings are subsequently fed into trans- former layers for text classification. However, these methods are not pre-trained. Inspired by this line of work, we propose HASHFORMERS, a new family of vocabulary-independent pre- trained transformers that support an unlimited vocabulary (i.e. all possible tokens in a cor- pus) given a substantially smaller fixed-sized embedding matrix. We achieve this by first in- troducing computationally cheap hashing func- tions that bucket together individual tokens to embeddings. We also propose three variants that do not require an embedding matrix at all, further reducing the memory requirements. We empirically demonstrate that HASHFORM- ERS are more memory efficient compared to standard pre-trained transformers while achiev- ing comparable predictive performance when fine-tuned on multiple text classification tasks. For example, our most efficient HASHFORMER variant has a negligible performance degrada- tion (0.4% on GLUE) using only 99.1K parame- ters for representing the embeddings compared to 12.3-38M parameters of state-of-the-art mod- els.1 1 Introduction The majority of transformer-based (Vaswani et al., 2017) pre-trained language models (PLMs; Devlin 1Code is available here: https://github.com/ HUIYINXUE/hashformer and the pre-traied HashFormer models are available here: https://huggingface.co/ klein9692. et al. 2019; Liu et al. 2019; Dai et al. 2019; Yang et al. 2019) are vocabulary-dependent, with each single token mapped to its corresponding vector in an embedding matrix. This one-to-one mapping makes it impractical to support out-of-vocabulary tokens such as misspellings or rare words (Pruthi et al., 2019; Sun et al., 2020). Moreover, it linearly increases the memory requirements with the vocab- ulary size for the token embedding matrix (Chung et al., 2021). For example, given a token embed- ding size of 768, BERT-BASE with a vocabulary of 30.5K tokens needs 23.4M out of 110M total pa- rameters while ROBERTA-BASE with 50K tokens needs 38M out of 125M total parameters. Hence, disentangling the design of PLMs from the vo- cabulary size and tokenization approaches would inherently improve memory efficiency and pre- training, especially for researchers with access to limited computing resources (Strubell et al., 2019; Schwartz et al., 2020). Previous efforts for making transformer-based models vocabulary-independent include dynam- ically generating token embeddings on-the-fly without embedding matrices using hash embed- dings (Svenstrup et al., 2017; Ravi, 2019) over morphological information (Sankar et al., 2021a). However, these embeddings are subsequently fed into transformer layers trained from scratch for on- device text classification …

Content truncated to 500 words