Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model

Proceedings of the 28th International Conference on Computational Linguistics, pages 6883–6893 Barcelona, Spain (Online), December 8-13, 2020 6883 Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model Sungrae Park∗ Junbum Cha∗ Geewook Kim∗ Ji-Hoon Kim∗,† Junyeop Lee∗,†† Hwalsuk Lee∗,†† ∗ CLOVA AI Research, NAVER Corporation † NAVER AI LAB, NAVER Corporation ††Upstage AI Research, Upstage {sungrae.park, gw.kim, junbum.cha, †genesis.kim}@navercorp.com ††{junyeop.lee, hwalsuk.lee}@upstage.ai Abstract This paper introduces a method that efficiently reduces the computational cost and parameter size of Transformer. The proposed model, refer to as Group-Transformer, splits feature space into multiple groups, factorizes the calculation paths, and reduces computations for the group interaction. Extensive experiments on two benchmark tasks, enwik8 and text8, prove our model’s effectiveness and efficiency in small-scale Transformers. To the best of our knowledge, Group- Transformer is the first attempt to design Transformer with the group strategy, widely used for efficient CNN architectures. 1 Introduction Character-level language modeling has become a core task in the field of natural language processing (NLP) such as classification (Zhang et al., 2015), sequence tagging (Guo et al., 2019a), question an- swering (He and Golub, 2016), and scene text recognition (Baek et al., 2019; Hwang and Sung, 2016), with its simplicity on generating text and its adaptability to other languages. Most previous approaches had consisted of recurrent neural networks (RNNs), but they have suffered from high learning com- plexity caused by inherently long character sequences. Recently, Transformer (Vaswani et al., 2017) have shown promise in addressing this problem and have become a standard way in general language modeling (Al-Rfou et al., 2019; Dai et al., 2019). Transformers have achieved higher performance but have also grown in size by building deeper and wider networks. TransformerXL (Dai et al., 2019) and GPT-2 (Radford et al., 2019), for instance, contain 277M and 1542M parameters, respectively. However, this trend toward a large size model for perfor- mance is not suitable for edge device applications that require small memory sizes and fast real-time responsiveness, such as auto-correction and auto-completion (Gong et al., 2019). Contrary to the recent trend, character-level language models need to be scaled down while minimizing performance degrada- tion due to capacity loss. A simple way to get a lightweight Transformer is to reduce its width and depth, directly related to the model complexity. However, the width reduction loses representation power of high dimensional feature space, and the depth reduction brings the lower capacity that stacks diverse dependencies between local information. To compensate for the losses, knowledge distillation (Sun et al., 2019) tries to optimize a scale-downed model with a large teacher network, and weight-sharing (Bai et al., 2019) stacks a unified layer multiple times. They have shown promising results, but they still require a scale-downed model as a target model, or a unified layer for iterative usage. In this paper, we introduce a lightweight Transformer, referred to as Group-Transformer, with lower model complexity without any modification to its width and depth. Our method utilizes group-wise operations, inspired by the group convolution approaches …

Content truncated to 500 words