Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6215–6224 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6215 Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units Arturo Argueta and David Chiang Department of Computer Science and Engineering University of Notre Dame {aargueta,dchiang}@nd.edu Abstract Graphics Processing Units (GPUs) are com- monly used to train and evaluate neural net- works efficiently. While previous work in deep learning has focused on accelerating opera- tions on dense matrices/tensors on GPUs, ef- forts have concentrated on operations involv- ing sparse data structures. Operations using sparse structures are common in natural lan- guage models at the input and output layers, because these models operate on sequences over discrete alphabets. We present two new GPU algorithms: one at the input layer, for multiplying a matrix by a few-hot vector (gen- eralizing the more common operation of mul- tiplication by a one-hot vector) and one at the output layer, for a fused softmax and top-N se- lection (commonly used in beam search). Our methods achieve speedups over state-of-the- art parallel GPU baselines of up to 7× and 50×, respectively. We also illustrate how our methods scale on different GPU architectures. 1 Introduction The speedups introduced by parallel architectures inspired the development of accelerators tailored towards specialized functions. Graphics Process- ing Units (GPUs) are now a standard platform for deep learning. GPUs provide faster model training and inference times compared to serial processors, because they can parallelize the linear algebra op- erations used so heavily in neural networks (Raina et al., 2009). Currently, major open source toolkits (Abadi et al., 2016) provide additional layers of abstrac- tion to support one or more parallel GPU architec- tures. The seamless compatibility with multiple GPUs allows researchers to train a single model on multiple hardware platforms with no signifi- cant changes to their code base and no specialized knowledge about the targeted architectures. The disadvantage of hardware agnostic APIs is the lack of optimizations for a set of task-specific func- tions. Adapting parallel neural operations to a spe- cific hardware platform is required to obtain op- timal speed. Since matrix operations are used heavily in deep learning, much research has been done on optimizing them on GPUs (Chetlur et al., 2014; Gupta et al., 2015). Recently, some efforts have been made to other kinds of operations: se- rial operations running on the GPU (Povey et al., 2016), operations not involving matrix multiplica- tions (Bogoychev et al., 2018), and models using sparse structures (Zhang et al., 2016). In this pa- per, we focus on sparse operations running exclu- sively on the GPU architecture. Much recent work in High Performance Com- puting (HPC) and Natural Language Processing (NLP) focuses on an expensive step of a model or models and optimizes it for a specific architecture. The lookup operation used in the input layer and the softmax function used in the output are two examples seen in machine translation, language modeling, and …

Content truncated to 500 words