Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 346 - 359 December 7-11, 2022 ©2022 Association for Computational Linguistics Linearizing Transformer with Key-Value Memory Yizhe Zhang∗ Meta AI † yizhe.zhang@hotmail.com Deng Cai∗ The Chinese University of Hong Kong thisisjcykcd@gmail.com Abstract Efficient transformer variants with linear time complexity have been developed to mitigate the quadratic computational overhead of the vanilla transformer. Among them are low- rank projection methods such as Linformer and kernel-based Transformers. Despite their unique merits, they usually suffer from a perfor- mance drop comparing with the vanilla trans- former on many sequence generation tasks, and often fail to obtain computation gain when the generation is short. We propose Mem- Sizer, an approach towards closing the per- formance gap while improving the efficiency even with short generation. It projects the source sequences into lower dimension rep- resentations like Linformer, while enjoying efficient recurrent-style incremental computa- tion similar to kernel-based transformers. This yields linear computation time and constant memory complexity at inference time. Mem- Sizer also employs a lightweight multi-head mechanism which renders the computation as light as a single-head model. We demon- strate that MemSizer provides an improved bal- ance between efficiency and accuracy over the vanilla transformer and other efficient trans- former variants in three typical sequence gen- eration tasks, including machine translation, abstractive text summarization, and language modeling. Our code is released at https: //github.com/jcyk/memsizer 1 Introduction Transformer (Vaswani et al., 2017) has become the de facto standard for almost all NLP tasks across the board. At the core of the vanilla transformer is the attention mechanism that captures the in- teractions between feature vectors at different po- sitions in a sequence. Despite its great success, the vanilla transformer models are typically com- putationally expensive as the computation of the ∗Equal contribution. † Currently at Apple attention mechanism scales quadratically with the sequence length. This bottleneck limits the effi- cient deployment of large-scale pre-trained models, such as GPT-3 (Brown et al., 2020), Image Trans- former (Parmar et al., 2018), Codex (Chen et al., 2021) and DALL-E (Ramesh et al., 2021). Training and deploying such gigantic transformer models can be prohibitively difficult for scenarios with limited resource budgets and may result in huge energy consumption and greenhouse gas emission (Strubell et al., 2019; Schwartz et al., 2020). A number of transformer variants have been proposed to reduce the computational overhead (Tay et al., 2020c). One family of methods lever- ages low-rank projections to reduce the number of pair-wise interactions (i.e., the size of attention matrices) (Wang et al., 2020; Xiong et al., 2021; Tay et al., 2020a). These methods first project the input sequence into a low-resolution represen- tation. For example, Wang et al. (2020) project the length dimension to a fixed feature dimension. Nevertheless, these methods have difficulties mod- eling variable-length sequences and autoregressive (causal) attention, impeding their applications in sequence generation tasks. Recent works propose to approximate the softmax attention through ker- nelization (Katharopoulos et al., 2020; Peng et al., 2021; …

Content truncated to 500 words