Mask Attention Networks: Rethinking and Strengthen Transformer

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692–1701 June 6–11, 2021. ©2021 Association for Computational Linguistics 1692 Mask Attention Networks: Rethinking and Strengthen Transformer Zhihao Fan1∗, Yeyun Gong2, Dayiheng Liu3, Zhongyu Wei1,6†, Siyuan Wang1, Jian Jiao4, Nan Duan2, Ruofei Zhang4, Xuanjing Huang5 1School of Data Science, Fudan University, China 2Microsoft Research Asia, 3DAMO Academy, 4Microsoft 5School of Computer Science, Fudan University, China 6Research Institute of Intelligent and Complex Systems, Fudan University, China {fanzh18,zywei,wangsy18,xjhuang}@fudan.edu.cn, liudayiheng.ldyh@alibaba-inc.com {yegong,Jian.Jiao,nanduan,bzhang}@microsoft.com Abstract Transformer is an attention-based neural net- work, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed- Forward Network (FFN). Existing research ex- plores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask ma- trices limit the capability for localness model- ing in text representation learning. We there- fore introduce a new layer named dynamic mask attention network (DMAN) with a learn- able mask matrix which is able to model lo- calness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a se- quential layered structure to combine the three types of layers. Extensive experiments on vari- ous tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer. 1 Introduction Recently, Transformer (Vaswani et al., 2017) has been widely applied in various natural lan- guage processing tasks, such as neural machine translation (Vaswani et al., 2017) and text sum- marization (Zhang et al., 2019). To further im- prove the performance of the text representation, Transformer-based variants have attracted a lot of attention (Lu et al., 2019; Sukhbaatar et al., 2019a,b; Bugliarello and Okazaki, 2019; Ma et al., 2020). Each building block of Transformer has two sublayers: Self-Attention Network (SAN) and Feed-Forward Network (FFN). Shaw et al. (2018) Asia.∗Work is done during internship at Microsoft Research †Corresponding author. Figure 1: The mask matrices of (a) SAN, (b) DMAN and (c) FFN in Mask Attention Networks. Color that fades from black to white means the values in mask matrices decrease from 1 to 0. presents an extension to SAN which incorpo- rates the relative positional information for the se- quence. Sukhbaatar et al. (2019a) proposes atten- tion span to control the maximum context size used in SAN and scales Transformer to long-range (∼ 8192 tokens) language modeling. Recently, some works targeting on FFN have been proposed. Lu et al. (2019) gives a new understanding of Trans- former from a multi-particle dynamic system point of view and designs a macaron architecture follow- ing Strang-Marchuk splitting scheme. Sukhbaatar et al. (2019b) regards the FFN as the persistent memory in SAN to augment SAN. These works focus on enhancing SAN or FFN, but neglect the inner relationship between SAN and FFN that hin- ders further improvement. …

Content truncated to 500 words