EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10786 - 10798 December 7-11, 2022 ©2022 Association for Computational Linguistics EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation Tao Ge Si-Qing Chen Furu Wei Microsoft {tage,sqchen,fuwei}@microsoft.com Abstract We introduce EDGEFORMER – a parameter- efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the pre- vious parameter-efficient Transformers, EDGE- FORMER applies two novel principles for cost- effective parameterization, allowing it to per- form better given the same parameter budget; moreover, EDGEFORMER is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers. Extensive experiments show EDGEFORMER can effectively outperform previous parameter- efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EDGELM1 – the pretrained version of EDGEFORMER, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice. 1 Introduction On-device modeling draws increasing attention for its unique advantages (Dhar et al., 2019). On the other hand, strict resource constraints prevent many neural networks performing well in the on-device setting. In Natural Language Processing (NLP), on-device sequence-to-sequence (seq2seq) genera- tion remains challenging, especially for the Trans- former (Vaswani et al., 2017) under strict resource constraints in both computation and memory. To customize the Transformer for seq2seq tasks in the on-device setting, we propose EDGE- FORMER – a novel parameter-efficient Trans- former of the encoder-decoder architecture. EDGE- FORMER is structurally similar to the standard Transformer with a deep encoder and shallow de- coder, but with a modification that it uses an in- 1https://github.com/microsoft/unilm/tree/ master/edgelm self-a'n cross-a'n FFN ReLU Wf2 Wf1 dffn d dffn d self-a'n cross-a'n ReLU Wf2 Wf1 dffn d dffn d FFN FFN (a) (b) Figure 1: (a) Vanilla Transformer decoder layer in which dffn > d; (b) Interleaved Transformer decoder layer with shared lightweight FFNs in which dffn < d. terleaved decoder with shared lightweight feed- forward networks, as shown in Figure 1. The mod- ified decoder architecture allows EDGEFORMER to apply two novel principles that we propose for cost-effective parameterization: 1) encoder-favored parameterization that suggests we parameterize the encoder using as many parameters as possible; 2) load-balanced parameterization that suggests we balance the load of model parameters to avoid them being either underused or overused in a neural net- work with shared parameterization. In addition to cost-effective parameterization, EDGEFORMER proposes and applies layer adap- tation to further improve the model with tied lay- ers, as Figure 2 shows. Inspired by parameter- efficient task transfer, we investigate 3 efficient layer adaptation approaches for improving the per- formance with negligible cost. We show EDGE- FORMER (with fewer than 10 million model param- eters) largely outperforms the strong UNIVERSAL TRANSFORMER baselines in the on-device setting with competitive results, and the int8-quantized EDGEFORMER can perform high-quality on-device seq2seq generation within around 100ms latency (20-30 sequence length on average) using …

Content truncated to 500 words