Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7056 - 7068 December 7-11, 2022 ©2022 Association for Computational Linguistics Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation Sunzhu Li1, Peng Zhang1∗, Guobing Gan1, Xiuqing Lu1, Benyou Wang2, Junqiu Wei3, Xin Jiang4 1College of Intelligence and Computing, Tianjin University, Tianjin, China 2School of Data Science, The Chinese University of Hong Kong, Shenzhen, China 3The Hong Kong Polytechnic University, China 4Huawei Noah’s Ark Lab, China {lisunzhu,pzhang,ganguobing,lvxiuqing}@tju.edu.cn junwei@polyu.edu.hk, wangbenyou@cuhk.edu.cn, Jiang.Xin@huawei.com Abstract Transformer has been demonstrated effective in Neural Machine Translation (NMT). However, it is memory-consuming and time-consuming in edge devices, resulting in some difficulties for real-time feedback. To compress and accel- erate Transformer, we propose a Hybrid Tensor- Train (HTT) decomposition, which retains full rank and meanwhile reduces operations and pa- rameters. A Transformer using HTT, named Hypoformer, consistently and notably outper- forms the recent lightweight SOTA methods on three standard translation tasks under different parameter and speed scales. In extreme low re- source scenarios, Hypoformer has a 7.1 point absolute improvement in BLEU and 1.27× speedup than the vanilla Transformer on the IWSLT’14 De-En task. 1 Introduction Transformer (Vaswani et al., 2017) is one of the most effective models in NMT. Transformer could achieve better performance by stacking more lay- ers and increasing the embedding dimension (Liu et al., 2020; Li et al., 2021a). However, this leads to a huge scale of parameters and a slow inference speed. Various works have been recently proposed to compress Transformer, such as light-weight vari- ants (Wu et al., 2020; Mehta et al., 2021), neural architecture search (Wang et al., 2020), parameter sharing (Reid et al., 2021), low-rank approximation (Ma et al., 2019; Liu et al., 2021; Hrinchuk et al., 2020), knowledge distillation (Kasai et al., 2020), layer reassignment (Hsu et al., 2020; Bérard et al., 2021; Li et al., 2021c). Following the line of low-rank approximation, existing works to compress NMT models (or re- lated deep Transformer models like BERT) 1) are specific to a part of sub-components, e.g., self- attention networks (Ma et al., 2019) or embedding layer (Hrinchuk et al., 2020); and 2) show limited ∗Corresponding Author or no potential for speedup during inference (Ma et al., 2019; Hrinchuk et al., 2020; Noach and Gold- berg, 2020; Liu et al., 2021). However, unlike the cloud-based system, it needs real-time feedback and faster inference in edge devices (e.g., smart- phones and IoTs) even with constrained resources. Although it is challenging, we believe that low-rank approximation has the potential for compressing and accelerating NMT models in edge devices. From a general point of view, this paper rethinks low-rank approximation methods, especially two representative methods namely Matrix Factoriza- tion (MF) and Tensor-Train decomposition (TT). The former is arguably time-efficient while the lat- ter is parameter-efficient. However, (1) MF will encounter a low-rank bottleneck with a high com- pression ratio, potentially leading to a moderate performance drop (Thakker et al., 2020). (2) Trans- former compression based on high TT-rank TT decelerates …

Content truncated to 500 words