Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2978 Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai⇤12, Zhilin Yang⇤12, Yiming Yang1, Jaime Carbonell1, Quoc V. Le2, Ruslan Salakhutdinov1 1Carnegie Mellon University, 2Google Brain {dzihang,zhiliny,yiming,jgc,rsalakhu}@cs.cmu.edu, qvl@google.com Abstract Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural ar- chitecture Transformer-XL that enables learn- ing dependency beyond a fixed length with- out disrupting temporal coherence. It con- sists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context frag- mentation problem. As a result, Transformer- XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Trans- formers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of- the-art results of bpc/perplexity to 0.99 on en- wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL man- ages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch1. 1 Introduction Language modeling is among the important prob- lems that require modeling long-term dependency, with successful applications such as unsupervised pretraining (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018). How- ever, it has been a challenge to equip neural networks with the capability to model long-term dependency in sequential data. Recurrent neu- ral networks (RNNs), in particular Long Short- ⇤Equal contribution. Order determined by swapping the one in Yang et al. (2017). 1https://github.com/kimiyoung/ transformer-xl Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997), have been a standard solu- tion to language modeling and obtained strong results on multiple benchmarks. Despite the wide adaption, RNNs are difficult to optimize due to gradient vanishing and explosion (Hochre- iter et al., 2001), and the introduction of gat- ing in LSTMs and the gradient clipping tech- nique (Graves, 2013) might not be sufficient to fully address this issue. Empirically, previous work has found that LSTM language models use 200 context words on average (Khandelwal et al., 2018), indicating room for further improvement. On the other hand, the direct connections be- tween long-distance word pairs baked in atten- tion mechanisms might ease optimization and en- able the learning of long-term dependency (Bah- danau et al., 2014; Vaswani et al., 2017). Re- cently, Al-Rfou et al. (2018) designed a set of aux- iliary losses to train deep Transformer networks for character-level language modeling, which out- perform LSTMs by a large margin. Despite the success, the LM training in Al-Rfou et al. (2018) is performed on separated fixed-length segments of a few hundred characters, …

Content truncated to 500 words