Hard-Coded Gaussian Attention for Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7689–7700 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7689 Hard-Coded Gaussian Attention for Neural Machine Translation Weiqiu You∗, Simeng Sun∗, Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst {wyou,simengsun,miyyer}@cs.umass.edu Abstract Recent work has questioned the importance of the Transformer’s multi-headed attention for achieving high translation quality. We push further in this direction by developing a “hard- coded” attention variant without any learned parameters. Surprisingly, replacing all learned self-attention heads in the encoder and decoder with fixed, input-agnostic Gaussian distribu- tions minimally impacts BLEU scores across four different language pairs. However, ad- ditionally hard-coding cross attention (which connects the decoder to the encoder) signifi- cantly lowers BLEU, suggesting that it is more important than self-attention. Much of this BLEU drop can be recovered by adding just a single learned cross attention head to an oth- erwise hard-coded Transformer. Taken as a whole, our results offer insight into which com- ponents of the Transformer are actually impor- tant, which we hope will guide future work into the development of simpler and more ef- ficient attention-based models. 1 Introduction The Transformer (Vaswani et al., 2017) has be- come the architecture of choice for neural machine translation. Instead of using recurrence to contextu- alize source and target token representations, Trans- formers rely on multi-headed attention mechanisms (MHA), which speed up training by enabling paral- lelization across timesteps. Recent work has called into question how much MHA contributes to trans- lation quality: for example, a significant fraction of attention heads in a pretrained Transformer can be pruned without appreciable loss in BLEU (Voita et al., 2019; Michel et al., 2019), and self-attention can be replaced by less expensive modules such as convolutions (Yang et al., 2018; Wu et al., 2019). In this paper, we take this direction to an ex- treme by developing a variant of MHA without * Authors contributed equally. Standard Transformer: scaled dot product of learned query and key vectors Jane went to the office Ours: fixed Gaussian distributions centered around nearby tokens Figure 1: Three heads of learned self-attention (top) as well as our hard-coded attention (bottom) given the query word “to”. In our variant, each attention head is a Gaussian distribution centered around a different token within a local window. any learned parameters (Section 3). Concretely, we replace each attention head with a “hard-coded” version, which is simply a standard normal distri- bution centered around a particular position in the sequence (Figure 1).1 When we replace all encoder and decoder self-attention mechanisms with our hard-coded variant, we achieve almost identical BLEU scores to the baseline Transformer for four different language pairs (Section 4).2 These experiments maintain fully learned MHA cross attention, which allows the decoder to con- dition its token representations on the encoder’s outputs. We next attempt to additionally replace cross attention with a hard-coded version, which re- sults in substantial drops of 5-10 BLEU. Motivated to find the …

Content truncated to 500 words