Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

Proceedings of Recent Advances in Natural Language Processing, pages 173–181 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_021 173 Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs? Kevin Blin Section of Communication Systems Bachelor semester 6 EPFL, Lausanne, Switzerland kevin.blin@epfl.ch Andrei Kucharavy Distributed Computing Laboratory EPFL, Lausanne, Switzerland andrei.kucharavy@epfl.ch Abstract In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversar- ial network (GAN) architecture - Diversity- Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in or- der to leverage their efficiency. The result- ing Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diver- sity of generated text and stability. Compu- tational experiments suggested that a trans- former architecture is unable to drop-in re- place the LSTM layer, under-performing dur- ing the pre-training phase and undergoing a complete mode collapse during the GAN tun- ing phase. Our results suggest that the trans- former architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs. 1 Introduction Since the introduction of the Transformer in late 2017 by Vaswani et al. (2017), pure self-attention architectures have become ubiquitous in the natural language processing community. Initially introduced by Bahdanau et al. (2015), the self-attention mechanism proved to be particu- larly useful in the context of neural machine transla- tion to augment existing recurrent neural networks (RNNs). Recurrent neural networks (RNNs), in- troduced by Rumelhart et al. (1986) and expanded by Hochreiter and Schmidhuber (1997); Cho et al. (2014), were the state of the art in learning and gen- erating sequential models, notably for texts. With the addition of the attention mechanism, in the con- text of machine translation, they could focus better on the words and word combinations correspond- ing to the same concepts in different languages and focus on learning equivalences between them as opposed to trying to infer them directly from the whole excerpts of parallel texts used for learning. Following the initial introduction of the atten- tion mechanism to augment the RNNs, architec- tures combining the two became de-facto state of the art. Stacking RNN layers, adding pass-through mechanisms and separating the architecture into encoder and decoder with an attention layer in the middle became a standard, powering among oth- ers Google’s Neural Machine Translation system (Wu et al., 2016; Johnson et al., 2017). Despite impressive performance, those architectures had a fundamental limitation to their ability to scale. The RNN training and evaluation are sequential by their nature, which means that architectures relying on the RNNs could hardly benefit from the arrival of massively parallel computing. The innovation of the Transformer was to show that it was possible to learn sequence-to-sequence mapping while dispensing entirely with RNN lay- ers, using only self-attention mechanisms (”Atten- tion is all you need”) - Vaswani et al. (2017)). By stacking several layers of self-attention networks to form an encoder and a …

Content truncated to 500 words