Quantization
Related Papers
Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation
AbstractThe deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead during inference, especially when the target device is limited in computational resources such …...
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
AbstractRecent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, label loyalty and …...
BinaryBERT: Pushing the Limit of BERT Quantization
AbstractThe rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, …...
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
How much information do NLP tasks really need from a transformer’s attention mecha- nism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that …...
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
AbstractTransformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and …...
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
AbstractQuantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their …...
A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models
AbstractCompression techniques for deep learning have become increasingly popular, particularly in settings where latency and memory constraints are imposed. Several methods, such as pruning, distillation, and quantization, have been adopted …...