Interpreting Positional Information in Perspective of Word Order

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9600–9613 July 9-14, 2023 ©2023 Association for Computational Linguistics Interpreting Positional Information in Perspective of Word Order Xilong Zhang1, Ruochen Liu1∗, Jin Liu1, Xuefeng Liang1,2 1School of Artificial Intelligence, Xidian University, Xi’an, China 2Guangzhou Institute of Technology, Xidian University, Guangzhou, China {xilongzhang,liujin_1}@stu.xidian.edu.cn {ruochenliu,xliang}@xidian.edu.cn Abstract The attention mechanism is a powerful and ef- fective method utilized in natural language pro- cessing. However, it has been observed that this method is insensitive to positional information. Although several studies have attempted to im- prove positional encoding and investigate the influence of word order perturbation, it remains unclear how positional encoding impacts NLP models from the perspective of word order. In this paper, we aim to shed light on this problem by analyzing the working mechanism of the at- tention module and investigating the root cause of its inability to encode positional information. Our hypothesis is that the insensitivity can be attributed to the weight sum operation utilized in the attention module. To verify this hypothe- sis, we propose a novel weight concatenation operation and evaluate its efficacy in neural ma- chine translation tasks. Our enhanced experi- mental results not only reveal that the proposed operation can effectively encode positional in- formation but also confirm our hypothesis. 1 Introduction In recent years, attention mechanism (Bahdanau et al., 2015; Luong et al., 2015; Lin et al., 2017) has made remarkable progress on a wide range of natural language processing (NLP) tasks, such as machine translation (Vaswani et al., 2017; Rad- ford et al., 2018), question answering and language inference (Devlin et al., 2019). It updates the con- textual representation of a word by aggregating information from other words in the context, en- abling it capture the content-based relevance of any two words, regardless of the distance between them. In contrast to widely used recurrent neural net- works (RNNs) or convolutional neural networks (CNNs), attention mechanism suffers from a no- table disadvantage, i.e. its permutation invari- ance (Yun et al., 2020). This limitation results in its ∗Corresponding author. �� � ∑ �� ⊙ (a) Weight Sum �� � �� ⊕ ⊙ (b) Weight Concatenation Figure 1: Illustration of calculating the contextual rep- resentation of xi with (a) weight sum (b) weight con- catenation in Transformer’s attention layers. ai is the attention score vector of xi over other words and V is the value matrix. ⊙, �, ⊕ are scalar multiplication, vector sum and vector concatenation respectively. The weight concatenation combines weighted elements in accordance with the word order and linearly transforms the output to its original dimensions. This process serves to establish positional dependencies throughout a given sequence. failure to distinguish sequences with different word orders. As a result, numerous studies have focused on encoding positional information of a sequence for the attention mechanism, such as learnable fixed-length positional encoding (Gehring et al., 2017), sinusoidal positional encoding (Vaswani et al., 2017) and relative positional encoding (Shaw et al., 2018). …

Content truncated to 500 words