Self-Attention with Structural Position Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1403–1409, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1403 Self-Attention with Structural Position Representations Xing Wang Tencent AI Lab brightxwang@tencent.com Zhaopeng Tu Tencent AI Lab zptu@tencent.com Longyue Wang Tencent AI Lab vinnylywang@tencent.com Shuming Shi Tencent AI Lab shumingshi@tencent.com Abstract Although self-attention networks (SANs) have advanced the state-of-the-art on various NLP tasks, one criticism of SANs is their ability of encoding positions of input words (Shaw et al., 2018). In this work, we propose to augment SANs with structural position rep- resentations to model the latent structure of the input sentence, which is complementary to the standard sequential positional represen- tations. Specifically, we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to en- code the positional relationships among words in the dependency tree. Experimental re- sults on NIST Chinese⇒English and WMT14 English⇒German translation tasks show that the proposed approach consistently boosts per- formance over both the absolute and relative sequential position representations. 1 Introduction In recent years, self-attention networks (SANs, Parikh et al., 2016; Lin et al., 2017) have achieved the state-of-the-art results on a variety of NLP tasks (Vaswani et al., 2017; Strubell et al., 2018; Devlin et al., 2019). SANs perform the atten- tion operation under the position-unaware “bag- of-words” assumption, in which positions of the input words are ignored. Therefore, absolute position (Vaswani et al., 2017) or relative posi- tion (Shaw et al., 2018) are generally used to cap- ture the sequential order of words in the sentence. However, several researches reveal that the se- quential structure may not be sufficient for NLP tasks (Tai et al., 2015; Kim et al., 2017; Shen et al., 2019), since sentences inherently have hierarchi- cal structures (Chomsky, 1965; Bever, 1970). In response to this problem, we propose to aug- ment SANs with structural position representa- tions to capture the hierarchical structure of the in- put sentence. The starting point for our approach is a recent finding: the latent structure of a sen- tence can be captured by structural depths and dis- tances (Hewitt and Manning, 2019). Accordingly, we propose absolute structural position to encode the depth of each word in a parsing tree, and rel- ative structural position to encode the distance of each word pair in the tree. giesWe implement our structural encoding strate- on top of TRANSFORMER (Vaswani et al., 2017) and conduct experiments on both NIST Chinese⇒English and WMT14 English⇒German translation tasks. Experi- mental results show that exploiting structural position encoding strategies consistently boosts performance over both the absolute and rela- tive sequential position representations across language pairs. Linguistic analyses (Conneau et al., 2018) reveal that the proposed structural position representation improves the translation performance with richer syntactic information. Our main contributions are: • Our study demonstrates the necessity and effectiveness of exploiting structural posi- tion encoding for SANs, which benefits from modeling syntactic …

Content truncated to 500 words