HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7675–7688 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7675 HAT: Hardware-Aware Transformers for Efficient Natural Language Processing Hanrui Wang1, Zhanghao Wu1, Zhijian Liu1, Han Cai1, Ligeng Zhu1, Chuang Gan2, Song Han1 1Massachusetts Institute of Technology, 2MIT-IBM Watson AI Lab {hanrui,zhwu,zhijian,hancai,ligeng,chuangg,songhan}@mit.edu Abstract Transformers are ubiquitous in Natural Lan- guage Processing (NLP) tasks, but they are dif- ficult to be deployed on hardware due to the in- tensive computation. To enable low-latency in- ference on resource-constrained hardware plat- forms, we propose to design Hardware-Aware Transformers (HAT) with neural architecture search. We first construct a large design space with arbitrary encoder-decoder attention and heterogeneous layers. Then we train a Super- Transformer that covers all candidates in the design space, and efficiently produces many SubTransformers with weight sharing. Fi- nally, we perform an evolutionary search with a hardware latency constraint to find a special- ized SubTransformer dedicated to run fast on the target hardware. Extensive experiments on four machine translation tasks demonstrate that HAT can discover efficient models for different hardware (CPU, GPU, IoT device). When running WMT’14 translation task on Raspberry Pi-4, HAT can achieve 3× speedup, 3.7× smaller size over baseline Transformer; 2.7× speedup, 3.6× smaller size over Evolved Transformer with 12,041× less search cost and no performance loss. HAT is open-sourced. 1 Introduction Transformer (Vaswani et al., 2017) has been widely used in natural language processing tasks. By stack- ing multiple identical encoder/decoder layers with attention modules, it provides a significant perfor- mance improvement over previous convolutional or recurrent neural network models (Kim, 2014). Nevertheless, it is challenging to deploy Trans- formers on mobile devices due to the high com- putation cost. For instance, in order to translate a sentence with only 30 words, a Transformer-Big model needs to execute 13G FLOPs and takes 20 seconds on a Raspberry Pi. Such long latency will hurt the user experience on edge devices. Thus we SuperTransformer Evolutionary Search with Hardware Constraints Specialized Deployment is Efficient Hardware Latency Feedback SubTransformers 12/3/2019 ����_�e����_1895988.��� ��e:///U�e��/�a�����a��/D�����ad�/����_�e����_1895988.��� 1/1 C�ea�ed b� Ca��lina Cani f��m �he N��n P��jec� IoT 12/3/2019 ����_g��_1132940.��g ��e:///U�e��/�a�����a�g/D�����ad�/����_g��_1132940.��g 1/1 C�ea�ed b� Mi�ha Pe��i�hche� f��m �he N��n P��jec� GPU 12/3/2019 Electronic De�ice �le:///Users/hanrui�ang/Do�nloads/noun_CPU_2880768.s�g 1/1 C�ea�ed b� M�ha��ad Kh�i��� A� f��� �he N��� P��jec� CPU Different Hardware 12/6/2019 ����_�a��b�a��_50324.��� ���:///U����/�a�����a��/D�����a��/����_�a��b�a��_50324.��� 1/1 C�ea�ed b� Da�id S�b��e��ki f��� �he N��� P��jec� TinyML Deeper Wider （Weight-Sharing） Figure 1: Framework for searching Hardware-Aware Transformers. We first train a SuperTransformer that contains numerous sub-networks, then conduct an evo- lutionary search with hardware latency feedback to find one specialized SubTransformer for each hardware. need hardware-efficient Transformers (Figure 1). There are two common pitfalls when evaluat- ing the efficiency of a Transformer. (1) FLOPs does not reflect the measured latency. Although FLOPs is used as an metric for efficiency in prior arts (Howard et al., 2017; Wu et al., 2020), it is not a good latency proxy. As in Figure 2 (Right), mod- els with the same FLOPs can …

Content truncated to 500 words