DA-Transformer: Distance-aware Transformer

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2059–2068 June 6–11, 2021. ©2021 Association for Computational Linguistics 2059 DA-Transformer: Distance-aware Transformer Chuhan Wu† Fangzhao Wu‡ Yongfeng Huang† †Department of Electronic Engineering & BNRist, Tsinghua University, Beijing 100084, China ‡Microsoft Research Asia, Beijing 100080, China {wuchuhan15,wufangzhao}@gmail.com, yfhuang@tsinghua.edu.cn Abstract Transformer has achieved great success in the NLP field by composing various advanced models like BERT and GPT. However, Trans- former and its existing variants may not be optimal in capturing token distances because the position or distance embeddings used by these methods usually cannot keep the precise information of real distances, which may not be beneficial for modeling the orders and re- lations of contexts. In this paper, we pro- pose DA-Transformer, which is a distance- aware Transformer that can exploit the real distance. We propose to incorporate the real distances between tokens to re-scale the raw self-attention weights, which are computed by the relevance between attention query and key. Concretely, in different self-attention heads the relative distance between each pair of to- kens is weighted by different learnable pa- rameters, which control the different prefer- ences on long- or short-term information of these heads. Since the raw weighted real dis- tances may not be optimal for adjusting self- attention weights, we propose a learnable sig- moid function to map them into re-scaled coef- ficients that have proper ranges. We first clip the raw self-attention weights via the ReLU function to keep non-negativity and introduce sparsity, and then multiply them with the re- scaled coefficients to encode real distance in- formation into self-attention. Extensive exper- iments on five benchmark datasets show that DA-Transformer can effectively improve the performance of many tasks and outperform the vanilla Transformer and its several variants. 1 Introduction Transformer (Vaswani et al., 2017) has achieved huge success in the NLP field in recent years (Kobayashi et al., 2020). It serves as the ba- sic architecture of various state-of-the-art models like BERT (Devlin et al., 2019) and GPT (Rad- ford et al., 2019), and boosts the performance of many tasks like text generation (Koncel-Kedziorski et al., 2019), machine translation (Vaswani et al., 2017), and reading comprehension (Xu et al., 2019). Thus, the improvement on the Transformer archi- tecture would be beneficial for many NLP-related fields (Wu et al., 2020a). A core component of Transformer is multi-head self-attention, which is responsible for modeling the relations between contexts (Yang et al., 2019; Guo et al., 2019). However, self-attention is position-agnostic since it does not distinguish the orders of inputs. Thus, in the vanilla Transformer, position encoding is applied to the input to help Transformer capture position information. How- ever, in contrast to recurrent and convolutional neu- ral networks, it is difficult for vanilla Transform- ers to be aware of the token distances (Shaw et al., 2018), which are usually important cues for context modeling. Thus, several works explored to incor- porate token distance information into Transformer. For example, Shaw …

Content truncated to 500 words