Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6887–6900 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6887 Highway Transformer: Self-Gating Enhanced Self-Attentive Networks Yekun Chai† Shuo Jin‡ Xinwen Hou† †Institute of Automation, Chinese Academy of Sciences ‡University of Pittsburgh chaiyekun@gmail.com shj42@pitt.edu Abstract Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various se- quence learning tasks, standing on the multi- headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic im- portance within the multi-dimensional latent space of individual representations. The sub- sidiary content-based SDU gates allow for the information flow of modulated latent embed- dings through skipped connections, leading to a clear margin of convergence speed with gra- dient descent algorithms. We may unveil the role of gating mechanism to aid in the context- based Transformer modules, with hypothesiz- ing that SDU gates, especially on shallow lay- ers, could push it faster to step towards subop- timal points during the optimization process. 1 Introduction Self-attention mechanism has lately attracted ex- tensive interests due to its remarkable achievement on a wide range of sequence modeling applications, including natural language processing such as neu- ral machine translation (Vaswani et al., 2017; Ott et al., 2018; Shaw et al., 2018), language model- ing (LM) (Dai et al., 2019; Al-Rfou et al., 2019), self-supervised pretraining (Radford et al., 2018; Devlin et al., 2018; Lan et al., 2019); image genera- tion (Parmar et al., 2018); deep reinforcement learn- ing (Zambaldi et al., 2018; Vinyals et al., 2019), etc. Holding the great promise of deep neural net- works in language and images, Transformer capi- talizes on the stacked multi-headed self-attention mechanism based on the conventional encoder- decoder architecture in a sequence-to-sequence (seq2seq) manner to learn the global soft signals without explicit recurrence mechanism. Multi-head dot product attention (MHDPA) not only underpins the parallel training of multiple heads but captures long-term dependencies across an arbitrarily long distance within the same context. In which sepa- rated multiple heads independently draw sub-level attentions within the latent semantic sub-space of a fixed dimension, where different heads are pre- sumed to signal different meaning aspects implic- itly (Vaswani et al., 2017). Additionally, residual connections between layers allow the deep tandem stack of multiple identical modules by impeding degradation problem during training (He et al., 2016). Thus Transformer architectures take the place of Recurrent Neural Networks (RNNs), es- pecially Long Short-Term Memory (LSTM) net- works (Hochreiter and Schmidhuber, 1997) to be the model solution to learning sequential data. Recently, there have been plenty of works con- tending that gating mechanisms could play a vital role or even entirely substitute RNNs or Transform- ers to model language sequences. Dauphin et al. (2017) firstly claimed that non-recurrent networks are also highly competitive with conventional RNN- dominated models in LM. They proposed the hi- erarchical gated temporal convolution neural net- …

Content truncated to 500 words