Token Dropping
method
ID: tokendropping
Related Papers
Token Dropping for Efficient BERT Pretraining
AbstractTransformer-based models generally allocate the same amount of computation for each token in a given sequence. We develop a simple but effective “token dropping” method to accelerate the pretraining of …...
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
AbstractToken dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at …...