Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1494–1503 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1494 Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks 1Yi Tay, 2Aston Zhang, 3Luu Anh Tuan, 4Jinfeng Rao∗, 5Shuai Zhang 6Shuohang Wang, 7Jie Fu, 8Siu Cheung Hui 1,8Nanyang Technological University, 2Amazon AI, 3MIT CSAIL 4Facebook AI, 5UNSW, 6Singapore Management University, 7Mila and Polytechnique Montr´eal ytay017@e.ntu.edu.sg Abstract Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural ar- chitectures for a potpourri of natural language processing (NLP) tasks. To this end, our mod- els exploit computation using Quaternion al- gebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly (75%) reduced parame- ter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architec- tures such as the Quaternion attention Model and Quaternion Transformer. Extensive exper- iments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to 75% reduction in pa- rameter size without significant loss in perfor- mance. 1 Introduction Neural network architectures such as Transform- ers (Vaswani et al., 2017; Dehghani et al., 2018) and attention networks (Parikh et al., 2016; Seo et al., 2016; Bahdanau et al., 2014) are dominant solutions in natural language processing (NLP) re- search today. Many of these architectures are pri- marily concerned with learning useful feature rep- resentations from data in which providing a strong architectural inductive bias is known to be ex- tremely helpful for obtaining stellar results. Unfortunately, many of these models are known to be heavily parameterized, with state-of-the-art models easily containing millions or billions of parameters (Vaswani et al., 2017; Radford et al., 2018; Devlin et al., 2018; Radford et al., 2019). This renders practical deployment challenging. As such, the enabling of efficient and lightweight ∗ Work done while at University of Maryland. adaptations of these models, without significantly degrading performance, would certainly have a positive impact on many real world applications. To this end, this paper explores a new way to improve/maintain the performance of these neural architectures while substantially reducing the pa- rameter cost (compression of up to 75%). In or- der to achieve this, we move beyond real space, exploring computation in Quaternion space (i.e., hypercomplex numbers) as an inductive bias. Hy- percomplex numbers comprise of a real and three imaginary components (e.g., i, j, k) in which inter- dependencies between these components are en- coded naturally during training via the Hamilton product ⊗. Hamilton products have fewer degrees of freedom, enabling up to four times compres- sion of model size. Technical details are deferred to subsequent sections. While Quaternion connectionist architectures have been considered in various deep learn- ing application areas such as speech recogni- tion (Parcollet et al., 2018b), kinematics/human motion (Pavllo et al., 2018) and computer vi- sion (Gaudet and …

Content truncated to 500 words