Assessing the Ability of Self-Attention Networks to Learn Word Order

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3635–3644 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3635 Assessing the Ability of Self-Attention Networks to Learn Word Order Baosong Yang† Longyue Wang‡ Derek F. Wong† Lidia S. Chao† Zhaopeng Tu‡∗ †NLP2CT Lab, Department of Computer and Information Science, University of Macau nlp2ct.baosong@gmail.com, {derekfw,lidiasc}@umac.mo ‡Tencent AI Lab {vinnylywang,zptu}@tencent.com Abstract Self-attention networks (SAN) have attracted a lot of interests due to their high parallelization and strong performance on a variety of NLP tasks, e.g. machine translation. Due to the lack of recurrence structure such as recurrent neural networks (RNN), SAN is ascribed to be weak at learning positional information of words for sequence modeling. However, nei- ther this speculation has been empirically con- firmed, nor explanations for their strong per- formances on machine translation tasks when “lacking positional information” have been ex- plored. To this end, we propose a novel word reordering detection task to quantify how well the word order information learned by SAN and RNN. Specifically, we randomly move one word to another position, and examine whether a trained model can detect both the original and inserted positions. Experimental results reveal that: 1) SAN trained on word re- ordering detection indeed has difficulty learn- ing the positional information even with the position embedding; and 2) SAN trained on machine translation learns better positional in- formation than its RNN counterpart, in which position embedding plays a critical role. Al- though recurrence structure make the model more universally-effective on learning word order, learning objectives matter more in the downstream tasks such as machine translation. 1 Introduction Self-attention networks (SAN, Parikh et al., 2016; Lin et al., 2017) have shown promising empirical results in a variety of natural language processing (NLP) tasks, such as machine translation (Vaswani et al., 2017), semantic role labelling (Strubell et al., 2018), and language representations (De- vlin et al., 2019). The popularity of SAN lies in ∗ Zhaopeng Tu is the corresponding author of the paper. This work was conducted when Baosong Yang was interning at Tencent AI Lab. its high parallelization in computation, and flexi- bility in modeling dependencies regardless of dis- tance by explicitly attending to all the signals. Po- sition embedding (Gehring et al., 2017) is gener- ally deployed to capture sequential information for SAN (Vaswani et al., 2017; Shaw et al., 2018). Recent studies claimed that SAN with position embedding is still weak at learning word order information, due to the lack of recurrence struc- ture that is essential for sequence modeling (Shen et al., 2018a; Chen et al., 2018; Hao et al., 2019). However, such claims are mainly based on a theo- retical argument, which have not been empirically validated. In addition, this can not explain well why SAN-based models outperform their RNN counterpart in machine translation – a benchmark sequence modeling task (Vaswani et al., 2017). Our goal in this work is to empirically assess the ability of SAN to learn word …

Content truncated to 500 words