HyperMixer: An MLP-based Low Cost Alternative to Transformers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15632–15654 July 9-14, 2023 ©2023 Association for Computational Linguistics HyperMixer: An MLP-based Low Cost Alternative to Transformers Florian Mai†♠ Arnaud Pannatier†♠ Fabio Fehr†♠ Haolin Chen†♠ François Marelli†♠ François Fleuret♣♠† James Henderson† †Idiap Research Institute, Martigny, Switzerland ♠EPFL, Lausanne, Switzerland ♣University of Geneva, Geneva, Switzerland Abstract Transformer-based architectures are the model of choice for natural language understanding, but they come at a significant cost, as they have quadratic complexity in the input length, re- quire a lot of training data, and can be difficult to tune. In the pursuit of lower costs, we investi- gate simple MLP-based architectures. We find that existing architectures such as MLPMixer, which achieves token mixing through a static MLP applied to each feature independently, are too detached from the inductive biases required for natural language understanding. In this pa- per, we propose a simple variant, HyperMixer, which forms the token mixing MLP dynami- cally using hypernetworks. Empirically, we demonstrate that our model performs better than alternative MLP-based models, and on par with Transformers. In contrast to Transformers, HyperMixer achieves these results at substan- tially lower costs in terms of processing time, training data, and hyperparameter tuning. 1 Introduction Attention-based architectures, such as the Trans- former (Vaswani et al., 2017), have accelerated the progress in many natural language understanding tasks. Part of their success is a result of a paralleliz- able training scheme over the input length. This improves training times and allows for larger vol- umes of data which makes these models amenable to pretraining (Radford et al., 2018; Devlin et al., 2019). Therefore, many current state-of-the-art models are fine-tuned extensions of large pretrained Transformers (Bommasani et al., 2021). However, these models come at a significant computational cost. They require considerable resources for pretraining and fine-tuning, which induces high energy consumption (Strubell et al., 2019) and limits access to research (Bommasani et al., 2021). Subsequently, Schwartz et al. (2020) argue the need for "Green AI". They propose a cost evaluation of a result R as following: Cost(R) ∝ E · D · H, where E is the computational cost measured in floating point operations (FPO) of a single exam- ple, D is the dataset size, and H is the number of hyperparameter configurations required during tuning. To achieve a cost reduction, this paper proposes a simpler alternative to Transformers. We take inspiration from the computer vision community, which has recently seen a surge of research on Multi-Layer Perceptrons (MLPs). Most promi- nently, MLPMixer (Tolstikhin et al., 2021), which is a simple architecture based on two MLPs: one for token mixing and one for feature mixing. How- ever, the token mixing MLP learns a fixed-size set of position-specific mappings, arguably making MLPMixer’s architecture too detached from the inductive biases needed for natural language under- standing, in contrast to Transformers (Henderson, 2020). In this paper, we propose a simple variant, Hy- perMixer (Figure 1), which creates a token mix- ing MLP …

Content truncated to 500 words