Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7975–7984, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7975 Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications Matthew Khoury*,1, Rumen Dangovski*,†,2, Longwu Ou1, Preslav Nakov3, Yichen Shen2, Li Jing†,1 (*) equal contribution, (†) work done at Lightelligence, Inc. 1Lightelligence, Inc. {matthew.khoury, longwu.ou, yichen, li}@lightelligence.ai 2Massachusetts Institute of Technology rumenrd@mit.edu 3Qatar Computing Research Institute, HBKU pnakov@hbku.edu.qa Abstract Deep neural networks have become the stan- dard approach to building reliable Natural Lan- guage Processing (NLP) applications, ranging from Neural Machine Translation (NMT) to di- alogue systems. However, improving accuracy by increasing the model size requires a large number of hardware computations, which can slow down NLP applications significantly at inference time. To address this issue, we propose a novel vector-vector-matrix architec- ture (VVMA), which greatly reduces the la- tency at inference time for NMT. This architec- ture takes advantage of specialized hardware that has low-latency vector-vector operations and higher-latency vector-matrix operations. It also reduces the number of parameters and FLOPs for virtually all models that rely on ef- ficient matrix multipliers without significantly impacting accuracy. We present empirical re- sults suggesting that our framework can re- duce the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. Finally, we show evidence suggesting that our VVMA extends to other domains, and we discuss novel hardware for its efficient use. 1 Introduction Artificial neural networks have become increas- ingly popular over the last decade as they excel in tasks such as object detection and speech recog- nition (LeCun et al., 2015), which are becoming more commonplace with the use of self-driving cars and virtual assistants. The rapid development of deep neural networks has also made them the dominant approach for natural language processing (NLP) applications, ranging from neural machine translation (NMT) (Bahdanau et al., 2015; Klein et al., 2017; Wu et al., 2016) and text summariza- tion (Rush et al., 2015; Nallapati et al., 2016; Liu et al., 2018) to virtual assistants such as Apple Siri, Amazon Alexa, and Google Home. matrix multiplication in hardware × ours: vector-vector-matrix (VVMA) single load fast load ⨀ × slow load vector-matrix (TPU) × requires breaking the matrix in pieces Figure 1: TPU vs. VVMA. Top: to multiply a vector by a matrix, the hardware tiles up the matrix. Bottom left: the TPU loads each piece. Bottom right: the VVMA loads a single piece (for broadcasting) and adds diago- nals for element-wise multiplication, which is faster. Unfortunately, neural networks are slow for train- ing, inference and use due to their vast computa- tional complexity. Several approaches have been proposed to address these issues including (a) quan- tization and pruning, (b) efficient models with less computational demand, and (c) specialized hard- ware accelerators (Sze et al., 2017). While direc- tion (a) has been well-studied (LeCun et al., 1990; Han et al., 2016b,a; Guo, 2018; Quinn and Balles- teros, 2018), and can be considered complementary to (b,c), …

Content truncated to 500 words