Transformers with Learnable Activation Functions

Findings of the Association for Computational Linguistics: EACL 2023, pages 2382–2398 May 2-6, 2023 ©2023 Association for Computational Linguistics Transformers with Learnable Activation Functions Haishuo Fang1 Ji-Ung Lee1 Nafise Sadat Moosavi1,2 Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt www.ukp.tu-darmstadt.de 2Department of Computer Science, The University of Sheffield Abstract Activation functions can have a significant im- pact on reducing the topological complexity of input data and therefore, improving a model’s performance. However, the choice of activa- tion functions is seldom discussed or explored in Transformer-based language models. As a common practice, commonly used activa- tion functions like Gaussian Error Linear Unit (GELU) are chosen beforehand and then re- main fixed from pre-training to fine-tuning. In this paper, we investigate the impact of activa- tion functions on Transformer-based models by utilizing rational activation functions (RAFs). In contrast to fixed activation functions (FAF), RAFs are capable of learning the optimal ac- tivation functions from data. Our experiments show that the RAF-based Transformer model (RAFT) achieves a better performance than its FAF-based counterpart (FAFT). For instance, we find that RAFT outperforms FAFT on the GLUE benchmark by 5.71 points when using only 100 training examples and by 2.05 points on SQuAD with all available data. Analyzing the shapes of the learned RAFs further unveils that they vary across different layers and dif- ferent tasks; opening a promising way to better analyze and understand large, pre-trained lan- guage models.1 1 Introduction Activation functions introduce non-linearity and increase neural networks’ representational capac- ity, and therefore, play an essential role in design- ing deep learning models (Nwankpa et al., 2018; Sharma et al., 2020; Dubey et al., 2022). Naitzat et al. (2020) explain the importance of activation functions by proposing to consider data as a topol- ogy with its own shape. They empirically show that activation functions accelerate the data topology transformation through different layers of a neu- ral network to simplify its complexity and make 1Code, models, and datasplits are available on GitHub https://github.com/UKPLab/2022-RAFT. it linearly separable in the output space. Their ex- periments show that choosing the right activation function can have a significant impact on the over- all performance. While any activation function can be used with Transformers (Vaswani et al., 2017), their choice is made before pre-training and remains fixed af- terwards. Hence, the inductive bias an activa- tion function imposes on the model cannot be ad- justed during pre-training or fine-tuning. As many Transformer-based models are pre-trained on a large amount of data, and changing the activation function for or during fine-tuning may negatively impact the performance2. Moreover, the simple case of finding the optimal combination of k differ- ent activation functions in n different feedforward layers results in kn possible combinations and be- comes intractable; e.g., 531,441 experiments for a 12-layer BERT model and three different acti- vation functions. As a result, most Transformer- based pre-trained models adopt the GELU activa- tion function that has been …

Content truncated to 500 words