Differentiable Window for Dynamic Local Attention

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6589–6599 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6589 Differentiable Window for Dynamic Local Attention Thanh-Tung Nguyen∗†¶, Xuan-Phi Nguyen∗†¶, Shafiq Joty¶§, Xiaoli Li† ¶Nanyang Technological University §Salesforce Research Asia †Institute for Infocomm Research, A-STAR Singapore {ng0155ng@e.;nguyenxu002@e.;srjoty@}ntu.edu.sg xlli@i2r.a-star.edu.sg Abstract We propose Differentiable Window, a new neu- ral module and general purpose component for dynamic window selection. While univer- sally applicable, we demonstrate a compelling use case of utilizing Differentiable Window to improve standard attention modules by en- abling more focused attentions over the input regions. We propose two variants of Differen- tiable Window, and integrate them within the Transformer architecture in two novel ways. We evaluate our proposed approach on a myr- iad of NLP tasks, including machine transla- tion, sentiment analysis, subject-verb agree- ment and language modeling. Our experimen- tal results demonstrate consistent and sizable improvements across all tasks. 1 Introduction Computing relative importance across a series of inputs can be regarded as one of the important ad- vances in modern deep learning research. This paradigm, commonly known as attention (Bah- danau et al., 2015), has demonstrated immense success across a wide spectrum of applications. To this end, learning to compute contextual representa- tions (Vaswani et al., 2017), to point to the relevant part in the input (Vinyals et al., 2015), or to select windows or spans (Wang and Jiang, 2017) from sequences forms the crux of many modern deep neural architectures. Despite aggressive advances in developing neu- ral modules for computing relative relevance (Lu- ong et al., 2015; Chiu and Raffel, 2018), there has been no general purpose solution for learning differ- entiable attention windows. While span selection- based pointer network models typically predict a start boundary and an end boundary (Wang and Jiang, 2017; Seo et al., 2017), these soft predic- tions generally reside at the last layer of the net- ∗*Equal contributions work and are softly optimized. To the best of our knowledge, there exists no general purpose com- ponent for learning differentiable windows within networks. Although the practical advantages of learning differentiable windows are plenty, this paper fo- cuses on improving attentions with differentiable windows. The key idea is to enable more focused attention, leveraging dynamic window selection for limiting (and guiding) the search space for the standard attention modules to work within. This can also be interpreted as performing a form of dynamic local attention. We make several key technical contributions. First, we formulate the dynamic window selec- tion problem as a problem of learning a discrete mask (i.e., binary values representing the window). By learning and composing left and right bound- aries, we show that we are able to parameterize the (discrete) masking method. We then propose soft adaptations of the above mentioned, namely trainable soft masking and segment-based soft masking, which are differentiable approximations that can not only be easily optimized in an end-to- end fashion, but also inherit the desirable properties of discrete masking. While these …

Content truncated to 500 words