Learn to Cross-lingual Transfer with Meta Graph Learning Across Heterogeneous Languages

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 236–255, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 236 Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference Bang An†, Jie Lyu†, Zhenyi Wang†, Chunyuan Li§ Changwei Hu∗, Fei Tan∗, Ruiyi Zhang‡, Yifan Hu∗, Changyou Chen† †State University of New York at Buffalo, §Microsoft Research, Redmond ∗Yahoo Research, ‡Duke University isbangan@gmail.com, ch237duke@gmail.com yifanhu@verizonmedia.com, changyou@buffalo.edu Abstract The neural attention mechanism plays an im- portant role in many natural language process- ing applications. In particular, multi-head at- tention extends single-head attention by allow- ing a model to jointly attend information from different perspectives. However, without ex- plicit constraining, multi-head attention may suffer from attention collapse, an issue that makes different heads extract similar atten- tive features, thus limiting the model’s repre- sentation power. In this paper, for the first time, we provide a novel understanding of multi-head attention from a Bayesian perspec- tive. Based on the recently developed particle- optimization sampling techniques, we propose a non-parametric approach that explicitly im- proves the repulsiveness in multi-head atten- tion and consequently strengthens model’s ex- pressiveness. Remarkably, our Bayesian inter- pretation provides theoretical inspirations on the not-well-understood questions: why and how one uses multi-head attention. Extensive experiments on various attention models and applications demonstrate that the proposed re- pulsive attention can improve the learned fea- ture diversity, leading to more informative rep- resentations with consistent performance im- provement on multiple tasks. 1 Introduction Multi-head attention (Vaswani et al., 2017) is an effective module in deep neural networks, with impressive performance gains in many natural- language-processing (NLP) tasks. By extending a single head to multiple paralleled attention heads, the architecture is widely adopted to capture dif- ferent attentive information and strengthen the ex- pressive power of a model. Lin et al. (2017) ap- plied the idea of multi-heads on self-attention and extract a 2-D matrix instead of a vector to repre- sent different contexts of a sentence. The Trans- former (Vaswani et al., 2017) and its variants such as BERT (Devlin et al., 2019) are influential ar- chitectures solely based on multi-head attention, achieving state-of-the-art performance on plenty of NLP tasks. The key of multi-head attention is its ability to jointly attend to information from differ- ent representation subspaces at different positions, which results in multiple latent features depicting the input data from different perspectives. How- ever, there are no explicit mechanisms guaranteeing this desired property, leading to potential attention redundancy or attention collapse, which has been observed in previous research (Voita et al., 2019; Kovaleva et al., 2019). Although there exist works by directly adding regularization on loss functions to encourage diversity in multi-head attention (Lin et al., 2017; Li et al., 2018), the underlying working principle has not been well-validated, and perfor- mance improvement is limited. Furthermore, an important problem on why and how multi-head at- tention improves over its single-head counterpart is poorly understood. In this paper, we provide a novel understanding of multi-head attention from a Bayesian perspective by …

Content truncated to 500 words