How Does Selective Mechanism Improve Self-Attention Networks?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2986–2995 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2986 How Does Selective Mechanism Improve Self-Attention Networks? Xinwei Geng1∗ Longyue Wang2 Xing Wang2 Bing Qin1,3 Ting Liu1,3 Zhaopeng Tu2 1Harbin Institute of Technology 2Tencent AI Lab 3Peng Cheng Laboratory 1{xwgeng, qinb, tliu}@ir.hit.edu.cn 2{vinnylywang, brightxwang, zptu}@tencent.com Abstract Self-attention networks (SANs) with selective mechanism has produced substantial improve- ments in various NLP tasks by concentrat- ing on a subset of input words. However, the underlying reasons for their strong per- formance have not been well explained. In this paper, we bridge the gap by assessing the strengths of selective SANs (SSANs), which are implemented with a flexible and univer- sal Gumbel-Softmax. Experimental results on several representative NLP tasks, includ- ing natural language inference, semantic role labelling, and machine translation, show that SSANs consistently outperform the standard SANs. Through well-designed probing exper- iments, we empirically validate that the im- provement of SSANs can be attributed in part to mitigating two commonly-cited weaknesses of SANs: word order encoding and structure modeling. Specifically, the selective mecha- nism improves SANs by paying more attention to content words that contribute to the meaning of the sentence. The code and data are released at https://github.com/xwgeng/SSAN. 1 Introduction Self-attention networks (SANs) (Lin et al., 2017) have achieved promising progress in various nat- ural language processing (NLP) tasks, including machine translation (Vaswani et al., 2017), natural language inference (Shen et al., 2018b), semantic role labeling (Tan et al., 2018; Strubell et al., 2018) and language representation (Devlin et al., 2019). The appealing strength of SANs derives from high parallelism as well as flexibility in modeling depen- dencies among all the input elements. Recently, there has been a growing interest in integrating selective mechanism into SANs, which has produced substantial improvements in a variety ∗ Work done when interning at Tencent AI Lab. of NLP tasks. For example, some researchers incor- porated a hard constraint into SANs to select a sub- set of input words, on top of which self-attention is conducted (Shen et al., 2018c; Hou et al., 2019; Yang et al., 2019b). Yang et al. (2018) and Guo et al. (2019) proposed a soft mechanism by imposing a learned Gaussian bias over the original attention distribution to enhance its ability of capturing local contexts. Shen et al. (2018c) incorporated rein- forced sampling to dynamically choose a subset of input elements, which are fed to SANs. Although the general idea of selective mecha- nism works well across NLP tasks, previous stud- ies only validate their own implementations in a few tasks, either on only classification tasks (Shen et al., 2018c; Guo et al., 2019) or sequence gen- eration tasks (Yang et al., 2018, 2019b). This poses a potential threat to the conclusive effective- ness of selective mechanism. In response to this problem, we adopt a flexible and universal imple- mentation of selective mechanism using Gumbel- Softmax (Jang et al., 2017), called selective self- attention networks …

Content truncated to 500 words